Jun 15 05:47:46.956664 ip-10-0-129-86 systemd[1]: Starting Kubernetes Kubelet...
Jun 15 05:47:47.378506 ip-10-0-129-86 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:47:47.378506 ip-10-0-129-86 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 15 05:47:47.378506 ip-10-0-129-86 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:47:47.378506 ip-10-0-129-86 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 15 05:47:47.378506 ip-10-0-129-86 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:47:47.379369 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.379281    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 15 05:47:47.383584 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383562    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:47.383584 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383583    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:47.383584 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383587    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383590    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383594    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383597    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383601    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383603    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383606    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383609    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383612    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383615    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383617    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383621    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383623    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383626    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383629    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383631    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383635    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383637    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383640    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383643    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:47.383700 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383645    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383648    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383650    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383653    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383657    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383660    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383663    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383666    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383668    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383671    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383674    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383676    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383679    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383681    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383685    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383687    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383692    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383695    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383699    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383702    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:47.384175 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383705    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383708    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383711    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383714    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383717    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383719    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383722    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383724    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383727    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383730    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383733    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383737    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383740    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383742    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383745    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383748    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383750    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383753    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383755    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383758    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:47.384724 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383760    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383763    2576 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383765    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383768    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383770    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383773    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383776    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383779    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383784    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383788    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383791    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383794    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383796    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383799    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383802    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383805    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383807    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383811    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383813    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383816    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:47.385206 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383818    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383821    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383823    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383826    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383828    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383831    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383833    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383836    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383838    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383841    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383843    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383846    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383848    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383851    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383853    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383856    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.383858    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385318    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385326    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385330    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:47.385795 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385334    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385336    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385340    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385343    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385346    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385349    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385352    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385355    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385357    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385359    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385362    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385365    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385368    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385370    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385374    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385377    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385380    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385383    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385385    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385388    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:47.386301 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385392    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385395    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385397    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385400    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385403    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385406    2576 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385408    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385411    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385414    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385416    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385419    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385421    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385424    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385427    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385429    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385433    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385435    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385438    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385440    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385443    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:47.386807 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385446    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385448    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385451    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385453    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385456    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385460    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385465    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385469    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385472    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385474    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385477    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385480    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385482    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385485    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385488    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385490    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385493    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385497    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385500    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:47.387334 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385502    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385505    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385507    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385510    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385512    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385515    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385517    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385520    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385523    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385526    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385528    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385531    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385534    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385537    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385540    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385542    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385545    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385547    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385550    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385554    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:47.387818 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385557    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385559    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385562    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385565    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385567    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385570    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385572    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385575    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385577    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385582    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385585    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385588    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385591    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385594    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385597    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385600    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.385603    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386052    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386059    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386063    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:47.388326 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386066    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386071    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386074    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386076    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386079    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386082    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386084    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386087    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386090    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386093    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386096    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386099    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386103    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386107    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386110    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386113    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386115    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386118    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386120    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:47.388847 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386123    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386126    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386129    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386131    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386134    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386136    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386139    2576 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386142    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386144    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386147    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386149    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386152    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386154    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386157    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386159    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386162    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386165    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386168    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386170    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386172    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:47.389317 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386175    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386178    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386182    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386185    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386188    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386191    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386194    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386196    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386199    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386201    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386204    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386207    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386210    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386212    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386215    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386217    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386220    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386222    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386225    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386229    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:47.389803 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386232    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386235    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386238    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386240    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386257    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386260    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386263    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386266    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386269    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386272    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386275    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386277    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386280    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386283    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386285    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386288    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386290    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386293    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386295    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386298    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:47.390345 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386300    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386303    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386306    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386309    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386311    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386315    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386318    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386321    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386324    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386326    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386329    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386336    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386339    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386341    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386344    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386347    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.386351    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386442    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386454    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386495    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386501    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 15 05:47:47.390841 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386507    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386510    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386515    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386520    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386523    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386526    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386530    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386534    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386537    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386540    2576 flags.go:64] FLAG: --cgroup-root=""
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386543    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386546    2576 flags.go:64] FLAG: --client-ca-file=""
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386549    2576 flags.go:64] FLAG: --cloud-provider="external"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386552    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386557    2576 flags.go:64] FLAG: --cluster-domain=""
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386560    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386563    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386567    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386570    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386574    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386577    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386580    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386586    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386589    2576 flags.go:64] FLAG: --contention-profiling="false"
Jun 15 05:47:47.391372 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386592    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386595    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386598    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386601    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386605    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386608    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386612    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386615    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386618    2576 flags.go:64] FLAG: --enable-server="true"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386622    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386626    2576 flags.go:64] FLAG: --event-burst="100"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386629    2576 flags.go:64] FLAG: --event-qps="50"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386632    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386636    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386639    2576 flags.go:64] FLAG: --eviction-hard=""
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386643    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386646    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386649    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386652    2576 flags.go:64] FLAG: --eviction-soft=""
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386655    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386658    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386660    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386664    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386666    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386669    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jun 15 05:47:47.391986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386672    2576 flags.go:64] FLAG: --feature-gates=""
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386676    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386679    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386682    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386686    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386690    2576 flags.go:64] FLAG: --healthz-port="10248"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386693    2576 flags.go:64] FLAG: --help="false"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386696    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386699    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386702    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386705    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386709    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386713    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386716    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386719    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386722    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386725    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386728    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386732    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386735    2576 flags.go:64] FLAG: --kube-reserved=""
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386738    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386741    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386744    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386747    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 15 05:47:47.392625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386750    2576 flags.go:64] FLAG: --lock-file=""
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386752    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386755    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386758    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386764    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386767    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386770    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386772    2576 flags.go:64] FLAG: --logging-format="text"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386775    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386779    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386782    2576 flags.go:64] FLAG: --manifest-url=""
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386785    2576 flags.go:64] FLAG: --manifest-url-header=""
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386789    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386793    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386798    2576 flags.go:64] FLAG: --max-pods="110"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386801    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386804    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386807    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386810    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386813    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386816    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386826    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386836    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386839    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386843    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 15 05:47:47.393221 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386846    2576 flags.go:64] FLAG: --pod-cidr=""
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386850    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a2e90c8ef7d783e4d635e24fb09435f5e1f6e954583512c0944f9e57bc77b52"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386856    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386859    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386862    2576 flags.go:64] FLAG: --pods-per-core="0"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386866    2576 flags.go:64] FLAG: --port="10250"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386869    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386872    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07d209cb4944739b0"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386875    2576 flags.go:64] FLAG: --qos-reserved=""
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386878    2576 flags.go:64] FLAG: --read-only-port="10255"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386881    2576 flags.go:64] FLAG: --register-node="true"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386884    2576 flags.go:64] FLAG: --register-with-taints=""
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386888    2576 flags.go:64] FLAG: --registry-burst="10"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386891    2576 flags.go:64] FLAG: --registry-qps="5"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386893    2576 flags.go:64] FLAG: --reserved-cpus=""
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386896    2576 flags.go:64] FLAG: --reserved-memory=""
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386900    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386904    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386907    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386910    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386912    2576 flags.go:64] FLAG: --runonce="false"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386916    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386920    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386923    2576 flags.go:64] FLAG: --seccomp-default="false"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386925    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 15 05:47:47.393847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386928    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386931    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386934    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386939    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386942    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386944    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386948    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386951    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386954    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386958    2576 flags.go:64] FLAG: --system-cgroups=""
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386961    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386967    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386970    2576 flags.go:64] FLAG: --tls-cert-file=""
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386973    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386978    2576 flags.go:64] FLAG: --tls-min-version=""
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386981    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386983    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386987    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386990    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386993    2576 flags.go:64] FLAG: --v="2"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.386997    2576 flags.go:64] FLAG: --version="false"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.387001    2576 flags.go:64] FLAG: --vmodule=""
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.387005    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.387008    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387137    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:47.394470 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387141    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387145    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387148    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387153    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387155    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387158    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387162    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387166    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387169    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387172    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387176    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387178    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387181    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387184    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387187    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387190    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387192    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387195    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387198    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387201    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:47.395112 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387204    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387206    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387209    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387212    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387215    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387217    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387220    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387222    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387225    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387228    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387231    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387233    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387236    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387239    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387255    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387260    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387263    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387266    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387269    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387272    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:47.395638 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387275    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387277    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387282    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387284    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387287    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387290    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387292    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387295    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387298    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387301    2576 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387304    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387306    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387309    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387312    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387315    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387317    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387320    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387322    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387325    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387327    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:47.396135 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387330    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387333    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387335    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387338    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387341    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387344    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387347    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387350    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387352    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387355    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387358    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387363    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387367    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387369    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387374    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387377    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387379    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387382    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387385    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:47.396684 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387387    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387391    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387394    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387411    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387416    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387420    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387423    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387425    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387428    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387431    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387434    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387437    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387440    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387442    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387445    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387448    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387450    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387453    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:47.397150 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.387456    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.388009    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.394414    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.394433    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.394454    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.394460    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394519    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394524    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394527    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394530    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394533    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394536    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394540    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394543    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394546    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:47.397600 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394549    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394552    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394554    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394557    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394560    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394563    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394566    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394568    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394571    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394573    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394576    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394578    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394581    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394583    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394586    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394589    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394593    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394597    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394600    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394603    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:47.397985 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394607    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394610    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394613    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394615    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394618    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394621    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394623    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394626    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394629    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394632    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394634    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394637    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394639    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394642    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394645    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394647    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394650    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394653    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394655    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394658    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:47.398489 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394661    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394665    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394668    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394672    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394674    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394677    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394679    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394682    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394685    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394687    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394690    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394693    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394696    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394700    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394702    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394705    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394708    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394711    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394713    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394716    2576 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:47.398965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394718    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394721    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394724    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394726    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394729    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394731    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394734    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394737    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394739    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394742    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394745    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394747    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394750    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394752    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394754    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394757    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394759    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394762    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394764    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394767    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:47.399479 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394770    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394774    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394778    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394781    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394785    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394788    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394793    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394796    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394799    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394801    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.394807    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394924    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394929    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394932    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394935    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:47.399974 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394938    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394941    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394943    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394946    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394949    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394952    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394954    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394957    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394960    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394962    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394966    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394971    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394974    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394976    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394979    2576 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394982    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394985    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394987    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394990    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.394992    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:47.400460 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395001    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395004    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395008    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395011    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395015    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395018    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395021    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395024    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395027    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395029    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395032    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395035    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395037    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395040    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395043    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395045    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395048    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395050    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395053    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:47.400975 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395055    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395058    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395060    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395063    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395065    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395068    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395070    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395073    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395076    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395078    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395081    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395084    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395086    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395089    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395092    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395094    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395097    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395118    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395123    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395126    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:47.401478 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395130    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395133    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395137    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395140    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395143    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395147    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395150    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395152    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395155    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395158    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395162    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395165    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395168    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395170    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395173    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395176    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395178    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395181    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395183    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:47.402053 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395186    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395190    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395192    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395195    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395197    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395200    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395203    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395206    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395209    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395211    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395214    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395216    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395220    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395223    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395225    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395228    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:47.402609 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:47.395230    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:47.403731 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.395235    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:47:47.403731 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.395385    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 15 05:47:47.403731 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.397456    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 15 05:47:47.403731 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.398704    2576 server.go:1019] "Starting client certificate rotation"
Jun 15 05:47:47.403731 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.399387    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 15 05:47:47.403731 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.399428    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 15 05:47:47.403731 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.401109    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 15 05:47:47.404503 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.404368    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 15 05:47:47.417699 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.417671    2576 log.go:25] "Validated CRI v1 runtime API"
Jun 15 05:47:47.420522 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.420507    2576 log.go:25] "Validated CRI v1 image API"
Jun 15 05:47:47.421968 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.421948    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 15 05:47:47.425501 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.425481    2576 fs.go:135] Filesystem UUIDs: map[22931f40-d49a-4626-88b1-64e0603cc2a4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ad1648ee-5d82-4271-b29c-7c7519bd8f3d:/dev/nvme0n1p3]
Jun 15 05:47:47.425559 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.425501    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 15 05:47:47.428367 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.428206    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 15 05:47:47.430929 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.430818    2576 manager.go:216] Machine: {Timestamp:2026-06-15 05:47:47.429556666 +0000 UTC m=+0.337437170 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3158573 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20500ed420cafbf582e59092e8425d SystemUUID:ec20500e-d420-cafb-f582-e59092e8425d BootID:94f76f93-1050-4077-89be-42f53b40d0e6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d4:2f:5b:0b:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d4:2f:5b:0b:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:6e:38:7b:96:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 15 05:47:47.430929 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.430928    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 15 05:47:47.431029 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.431018    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.120.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260608-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 15 05:47:47.433813 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.433790    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 15 05:47:47.433995 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.433818    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-86.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 15 05:47:47.434527 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.434517    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jun 15 05:47:47.434562 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.434530    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jun 15 05:47:47.434562 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.434537    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 15 05:47:47.435197 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.435187    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 15 05:47:47.436394 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.436385    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jun 15 05:47:47.439177 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.439166    2576 state_mem.go:36] "Initialized new in-memory state store"
Jun 15 05:47:47.439308 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.439298    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jun 15 05:47:47.439342 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.439336    2576 kubelet.go:486] "Attempting to sync node with API server"
Jun 15 05:47:47.439373 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.439350    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 15 05:47:47.439373 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.439366    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 15 05:47:47.439428 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.439376    2576 kubelet.go:398] "Adding apiserver pod source"
Jun 15 05:47:47.439428 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.439386    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 15 05:47:47.439736 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.439719    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 15 05:47:47.439736 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.439738    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 15 05:47:47.442998 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.442976    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jun 15 05:47:47.444414 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.444397    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 15 05:47:47.444514 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.444437    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jun 15 05:47:47.444636 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.444623    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 15 05:47:47.444698 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.444643    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 15 05:47:47.444698 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.444652    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 15 05:47:47.444698 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.444662    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 15 05:47:47.444698 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.444676    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 15 05:47:47.444698 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.444685    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 15 05:47:47.445042 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.445032    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 15 05:47:47.445089 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.445045    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 15 05:47:47.445089 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.445057    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 15 05:47:47.445089 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.445066    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 15 05:47:47.445089 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.445078    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 15 05:47:47.445271 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.445090    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 15 05:47:47.445832 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.445820    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 15 05:47:47.445877 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.445835    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 15 05:47:47.449812 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.449791    2576 server.go:1268] "Started kubelet"
Jun 15 05:47:47.449923 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.449871    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 15 05:47:47.450231 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.450204    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 15 05:47:47.450231 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.450208    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 15 05:47:47.450231 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.450191    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 15 05:47:47.450442 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.450282    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jun 15 05:47:47.450442 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.450325    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-86.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 15 05:47:47.450830 ip-10-0-129-86 systemd[1]: Started Kubernetes Kubelet.
Jun 15 05:47:47.451943 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.451930    2576 server.go:310] "Adding debug handlers to kubelet server"
Jun 15 05:47:47.452008 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.451980    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 15 05:47:47.453123 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.453105    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-54zj6"
Jun 15 05:47:47.455342 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.454020    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-86.ec2.internal.18b92b0e2ad31b35  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-86.ec2.internal,UID:ip-10-0-129-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-86.ec2.internal,},FirstTimestamp:2026-06-15 05:47:47.449756469 +0000 UTC m=+0.357636973,LastTimestamp:2026-06-15 05:47:47.449756469 +0000 UTC m=+0.357636973,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-86.ec2.internal,}"
Jun 15 05:47:47.456414 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.456396    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 15 05:47:47.456508 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.456469    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 15 05:47:47.456508 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.456497    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:47.457112 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457093    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jun 15 05:47:47.457192 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457095    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jun 15 05:47:47.457192 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457127    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jun 15 05:47:47.457323 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457225    2576 reconstruct.go:97] "Volume reconstruction finished"
Jun 15 05:47:47.457323 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457234    2576 reconciler.go:29] "Reconciler: start to sync state"
Jun 15 05:47:47.457461 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457444    2576 factory.go:55] Registering systemd factory
Jun 15 05:47:47.457461 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457464    2576 factory.go:223] Registration of the systemd container factory successfully
Jun 15 05:47:47.457618 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.457597    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 15 05:47:47.457786 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457770    2576 factory.go:153] Registering CRI-O factory
Jun 15 05:47:47.457786 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457789    2576 factory.go:223] Registration of the crio container factory successfully
Jun 15 05:47:47.457911 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457840    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 15 05:47:47.457911 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457864    2576 factory.go:103] Registering Raw factory
Jun 15 05:47:47.457911 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.457879    2576 manager.go:1196] Started watching for new ooms in manager
Jun 15 05:47:47.458223 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.458209    2576 manager.go:349] Starting recovery of all containers
Jun 15 05:47:47.463852 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.463017    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-54zj6"
Jun 15 05:47:47.464794 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.464768    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jun 15 05:47:47.464936 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.464906    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jun 15 05:47:47.468209 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.468188    2576 manager.go:354] Recovery completed
Jun 15 05:47:47.473346 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.473333    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:47.476750 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.476519    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:47.476750 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.476553    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:47.476750 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.476583    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:47.477571 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.477556    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 15 05:47:47.477571 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.477571    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 15 05:47:47.477669 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.477589    2576 state_mem.go:36] "Initialized new in-memory state store"
Jun 15 05:47:47.477669 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.477635    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:47.478557 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.478541    2576 policy_none.go:49] "None policy: Start"
Jun 15 05:47:47.478557 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.478558    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jun 15 05:47:47.478667 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.478571    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jun 15 05:47:47.478667 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.478598    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-86.ec2.internal.18b92b0e2c6bc9a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-86.ec2.internal,UID:ip-10-0-129-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-86.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-86.ec2.internal,},FirstTimestamp:2026-06-15 05:47:47.47653981 +0000 UTC m=+0.384420315,LastTimestamp:2026-06-15 05:47:47.47653981 +0000 UTC m=+0.384420315,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-86.ec2.internal,}"
Jun 15 05:47:47.479430 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.479418    2576 policy_none.go:47] "Start"
Jun 15 05:47:47.479900 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.479884    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:47.479957 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.479915    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:47.479957 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.479928    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:47.514922 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.514903    2576 manager.go:337] "Starting Device Plugin manager"
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.514964    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.514983    2576 server.go:87] "Starting device plugin registration server"
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.515263    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.515278    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.515369    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.515459    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.515468    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.516038    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.516078    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:47.526721 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.525641    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jun 15 05:47:47.527122 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.526995    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jun 15 05:47:47.527122 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.527028    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jun 15 05:47:47.527122 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.527050    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jun 15 05:47:47.527122 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.527085    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 15 05:47:47.529743 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.529720    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:47.615445 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.615410    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:47.616627 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.616607    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:47.616722 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.616641    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:47.616722 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.616651    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:47.616722 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.616674    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.625713 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.625695    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.625799 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.625721    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-86.ec2.internal\": node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:47.627946 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.627929    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal"]
Jun 15 05:47:47.627992 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.627988    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:47.629292 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.629234    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:47.629292 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.629274    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:47.629292 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.629288    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:47.630337 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.630324    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:47.630448 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.630427    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.630486 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.630465    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:47.630993 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.630973    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:47.630993 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.630981    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:47.631130 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.631004    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:47.631130 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.631007    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:47.631130 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.631014    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:47.631130 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.631018    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:47.632007 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.631989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.632089 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.632014    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:47.632621 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.632604    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:47.632743 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.632630    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:47.632743 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.632640    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:47.644401 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.644382    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:47.659030 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.659000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cbc9805fb56ac37d1dbd6954f1bb4020-config\") pod \"kube-apiserver-proxy-ip-10-0-129-86.ec2.internal\" (UID: \"cbc9805fb56ac37d1dbd6954f1bb4020\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.659147 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.659055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc9ec5d46dd6686220f17504d67a7115-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal\" (UID: \"fc9ec5d46dd6686220f17504d67a7115\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.659147 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.659084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc9ec5d46dd6686220f17504d67a7115-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal\" (UID: \"fc9ec5d46dd6686220f17504d67a7115\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.659269 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.659228    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-86.ec2.internal\" not found" node="ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.663910 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.663892    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-86.ec2.internal\" not found" node="ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.745207 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.745176    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:47.760185 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.760149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cbc9805fb56ac37d1dbd6954f1bb4020-config\") pod \"kube-apiserver-proxy-ip-10-0-129-86.ec2.internal\" (UID: \"cbc9805fb56ac37d1dbd6954f1bb4020\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.760185 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.760184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc9ec5d46dd6686220f17504d67a7115-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal\" (UID: \"fc9ec5d46dd6686220f17504d67a7115\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.760422 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.760203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc9ec5d46dd6686220f17504d67a7115-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal\" (UID: \"fc9ec5d46dd6686220f17504d67a7115\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.760422 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.760264    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cbc9805fb56ac37d1dbd6954f1bb4020-config\") pod \"kube-apiserver-proxy-ip-10-0-129-86.ec2.internal\" (UID: \"cbc9805fb56ac37d1dbd6954f1bb4020\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.760422 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.760271    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc9ec5d46dd6686220f17504d67a7115-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal\" (UID: \"fc9ec5d46dd6686220f17504d67a7115\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.760422 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.760267    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc9ec5d46dd6686220f17504d67a7115-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal\" (UID: \"fc9ec5d46dd6686220f17504d67a7115\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.845746 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.845716    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:47.946573 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:47.946516    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:47.961870 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.961835    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:47.966924 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:47.966903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:48.047537 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:48.047496    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:48.148013 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:48.147981    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:48.248594 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:48.248509    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:48.349105 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:48.349073    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:48.398590 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.398564    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 15 05:47:48.399191 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.398711    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 15 05:47:48.404562 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:48.404520    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9ec5d46dd6686220f17504d67a7115.slice/crio-b7e9550e265fd93a87ef6e1e94be179ec7ab6678153b28988e014e4bd7398928 WatchSource:0}: Error finding container b7e9550e265fd93a87ef6e1e94be179ec7ab6678153b28988e014e4bd7398928: Status 404 returned error can't find the container with id b7e9550e265fd93a87ef6e1e94be179ec7ab6678153b28988e014e4bd7398928
Jun 15 05:47:48.404995 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:48.404975    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc9805fb56ac37d1dbd6954f1bb4020.slice/crio-cc44b9186fba8f5836eb7bdf53642562d85a55dc04f42e3746a2b17abf790d30 WatchSource:0}: Error finding container cc44b9186fba8f5836eb7bdf53642562d85a55dc04f42e3746a2b17abf790d30: Status 404 returned error can't find the container with id cc44b9186fba8f5836eb7bdf53642562d85a55dc04f42e3746a2b17abf790d30
Jun 15 05:47:48.410778 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.410760    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 05:47:48.449713 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:48.449675    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:48.467459 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.467421    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-14 05:42:47 +0000 UTC" deadline="2028-03-16 02:20:03.115947672 +0000 UTC"
Jun 15 05:47:48.467459 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.467450    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15356h32m14.648500202s"
Jun 15 05:47:48.526103 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.526028    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:48.530415 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.530366    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal" event={"ID":"fc9ec5d46dd6686220f17504d67a7115","Type":"ContainerStarted","Data":"b7e9550e265fd93a87ef6e1e94be179ec7ab6678153b28988e014e4bd7398928"}
Jun 15 05:47:48.531410 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.531388    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal" event={"ID":"cbc9805fb56ac37d1dbd6954f1bb4020","Type":"ContainerStarted","Data":"cc44b9186fba8f5836eb7bdf53642562d85a55dc04f42e3746a2b17abf790d30"}
Jun 15 05:47:48.550596 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:48.550553    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-86.ec2.internal\" not found"
Jun 15 05:47:48.572574 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.572549    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:48.650278 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.650229    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:48.657287 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.657262    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:48.665521 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.665504    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 15 05:47:48.666407 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.666393    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal"
Jun 15 05:47:48.680685 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:48.680659    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 15 05:47:49.312340 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.312308    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:49.441107 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.441064    2576 apiserver.go:52] "Watching apiserver"
Jun 15 05:47:49.448799 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.448774    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 15 05:47:49.449817 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.449791    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-xf8tc","openshift-multus/network-metrics-daemon-cfxz2","openshift-network-diagnostics/network-check-target-wsmkh","kube-system/konnectivity-agent-mtz4k","kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp","openshift-image-registry/node-ca-5k6jr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal","openshift-network-operator/iptables-alerter-zk5nh","openshift-ovn-kubernetes/ovnkube-node-gb7k9","kube-system/global-pull-secret-syncer-2czf6","openshift-cluster-node-tuning-operator/tuned-sckf9","openshift-dns/node-resolver-pspvw","openshift-multus/multus-additional-cni-plugins-f2s9g"]
Jun 15 05:47:49.451128 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.451102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.452172 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.452149    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:49.452281 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.452233    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:47:49.453102 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.453080    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:49.453195 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.453150    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:47:49.454049 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.454029    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 15 05:47:49.454137 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.454098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:47:49.454137 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.454132    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 15 05:47:49.454318 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.454302    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j44n5\""
Jun 15 05:47:49.454390 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.454357    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 15 05:47:49.455175 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.455158    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.456625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.456368    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.456625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.456402    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 15 05:47:49.456625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.456462    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 15 05:47:49.456625 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.456502    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xvb9s\""
Jun 15 05:47:49.457006 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.456991    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 15 05:47:49.457700 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.457430    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.457978 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.457830    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 15 05:47:49.457978 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.457916    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 15 05:47:49.457978 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.457929    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hxqxt\""
Jun 15 05:47:49.457978 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.457918    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 15 05:47:49.457978 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.457961    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 15 05:47:49.458203 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.458041    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 15 05:47:49.458727 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.458699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.458818 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.458752    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 15 05:47:49.458818 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.458781    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 15 05:47:49.461836 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.461813    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqxck\""
Jun 15 05:47:49.462075 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.462063    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 15 05:47:49.462289 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.462268    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lq6jc\""
Jun 15 05:47:49.462631 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.462612    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 15 05:47:49.462732 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.458706    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 15 05:47:49.463099 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.463079    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 15 05:47:49.464063 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.463482    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 15 05:47:49.464936 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.464911    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:49.465028 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.464942    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6vbs\""
Jun 15 05:47:49.465028 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.464990    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 15 05:47:49.465028 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.465007    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:47:49.466367 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.466347    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.467691 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.467672    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.468548 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.468524    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 15 05:47:49.468899 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.468741    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 15 05:47:49.468993 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.468972    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.469203 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.469185    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 15 05:47:49.469464 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.469444    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 15 05:47:49.469581 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.469556    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 15 05:47:49.470052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-system-cni-dir\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.470154 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:49.470154 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470092    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-registration-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.470320 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecd20280-4ff2-42de-b4e2-85cdf88de891-host\") pod \"node-ca-5k6jr\" (UID: \"ecd20280-4ff2-42de-b4e2-85cdf88de891\") " pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.470320 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-cnibin\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.470320 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-os-release\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.470320 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470338    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470384    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8vkgw\""
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470396    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-kubelet-config\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470400    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mhbb8\""
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470470    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplgn\" (UniqueName: \"kubernetes.io/projected/5ddb948a-df3c-4979-8e17-c33ac7763f34-kube-api-access-gplgn\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-sys-fs\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4hcp\" (UniqueName: \"kubernetes.io/projected/ecd20280-4ff2-42de-b4e2-85cdf88de891-kube-api-access-m4hcp\") pod \"node-ca-5k6jr\" (UID: \"ecd20280-4ff2-42de-b4e2-85cdf88de891\") " pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-dbus\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:49.470534 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecd20280-4ff2-42de-b4e2-85cdf88de891-serviceca\") pod \"node-ca-5k6jr\" (UID: \"ecd20280-4ff2-42de-b4e2-85cdf88de891\") " pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19d542ed-18cc-46f9-bf7a-959f70dd7077-agent-certs\") pod \"konnectivity-agent-mtz4k\" (UID: \"19d542ed-18cc-46f9-bf7a-959f70dd7077\") " pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-device-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19d542ed-18cc-46f9-bf7a-959f70dd7077-konnectivity-ca\") pod \"konnectivity-agent-mtz4k\" (UID: \"19d542ed-18cc-46f9-bf7a-959f70dd7077\") " pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470699    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a3e4383-5637-45fd-b49d-9d77f7573feb-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a3e4383-5637-45fd-b49d-9d77f7573feb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a3e4383-5637-45fd-b49d-9d77f7573feb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb7bb954-4346-4175-8721-5ffbab5ef3d8-host-slash\") pod \"iptables-alerter-zk5nh\" (UID: \"bb7bb954-4346-4175-8721-5ffbab5ef3d8\") " pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7plm\" (UniqueName: \"kubernetes.io/projected/bb7bb954-4346-4175-8721-5ffbab5ef3d8-kube-api-access-g7plm\") pod \"iptables-alerter-zk5nh\" (UID: \"bb7bb954-4346-4175-8721-5ffbab5ef3d8\") " pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470838    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chcvn\" (UniqueName: \"kubernetes.io/projected/3a3e4383-5637-45fd-b49d-9d77f7573feb-kube-api-access-chcvn\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb7bb954-4346-4175-8721-5ffbab5ef3d8-iptables-alerter-script\") pod \"iptables-alerter-zk5nh\" (UID: \"bb7bb954-4346-4175-8721-5ffbab5ef3d8\") " pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.470905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-socket-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.471121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.471073    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 15 05:47:49.471945 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.471400    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 15 05:47:49.471945 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.471734    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ldl5\""
Jun 15 05:47:49.494471 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.494431    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nkhhm"
Jun 15 05:47:49.504410 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.504390    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nkhhm"
Jun 15 05:47:49.558877 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.558827    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jun 15 05:47:49.571541 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-run-k8s-cni-cncf-io\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.571541 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-cni-netd\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.571541 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571520    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8q6j\" (UniqueName: \"kubernetes.io/projected/1739a3c1-1f89-4206-b475-4c4b7374c36d-kube-api-access-f8q6j\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571547    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-kubelet-config\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt8dv\" (UniqueName: \"kubernetes.io/projected/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-kube-api-access-vt8dv\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplgn\" (UniqueName: \"kubernetes.io/projected/5ddb948a-df3c-4979-8e17-c33ac7763f34-kube-api-access-gplgn\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-sys-fs\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4hcp\" (UniqueName: \"kubernetes.io/projected/ecd20280-4ff2-42de-b4e2-85cdf88de891-kube-api-access-m4hcp\") pod \"node-ca-5k6jr\" (UID: \"ecd20280-4ff2-42de-b4e2-85cdf88de891\") " pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d79a647-5223-4980-8d70-40168d1c14fb-cni-binary-copy\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571639    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-kubelet-config\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-modprobe-d\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571766    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-sys-fs\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.571791 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-sysconfig\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-sys\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-dbus\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecd20280-4ff2-42de-b4e2-85cdf88de891-serviceca\") pod \"node-ca-5k6jr\" (UID: \"ecd20280-4ff2-42de-b4e2-85cdf88de891\") " pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19d542ed-18cc-46f9-bf7a-959f70dd7077-agent-certs\") pod \"konnectivity-agent-mtz4k\" (UID: \"19d542ed-18cc-46f9-bf7a-959f70dd7077\") " pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52ce0a05-4c82-4b03-8386-6ebdb0a6bae5-hosts-file\") pod \"node-resolver-pspvw\" (UID: \"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5\") " pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571953    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-var-lib-cni-multus\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.571995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-cni-bin\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52951973-c105-489a-9d13-b213aff03d3d-ovnkube-config\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-device-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19d542ed-18cc-46f9-bf7a-959f70dd7077-konnectivity-ca\") pod \"konnectivity-agent-mtz4k\" (UID: \"19d542ed-18cc-46f9-bf7a-959f70dd7077\") " pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572046    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-dbus\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a3e4383-5637-45fd-b49d-9d77f7573feb-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a3e4383-5637-45fd-b49d-9d77f7573feb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a3e4383-5637-45fd-b49d-9d77f7573feb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-var-lib-cni-bin\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-var-lib-openvswitch\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.572229 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572216    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7plm\" (UniqueName: \"kubernetes.io/projected/bb7bb954-4346-4175-8721-5ffbab5ef3d8-kube-api-access-g7plm\") pod \"iptables-alerter-zk5nh\" (UID: \"bb7bb954-4346-4175-8721-5ffbab5ef3d8\") " pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chcvn\" (UniqueName: \"kubernetes.io/projected/3a3e4383-5637-45fd-b49d-9d77f7573feb-kube-api-access-chcvn\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-sysctl-conf\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-run\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb7bb954-4346-4175-8721-5ffbab5ef3d8-iptables-alerter-script\") pod \"iptables-alerter-zk5nh\" (UID: \"bb7bb954-4346-4175-8721-5ffbab5ef3d8\") " pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572348    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572360    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-run-netns\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-node-log\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572405    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-device-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52951973-c105-489a-9d13-b213aff03d3d-ovnkube-script-lib\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecd20280-4ff2-42de-b4e2-85cdf88de891-host\") pod \"node-ca-5k6jr\" (UID: \"ecd20280-4ff2-42de-b4e2-85cdf88de891\") " pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-os-release\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjjdr\" (UniqueName: \"kubernetes.io/projected/8d79a647-5223-4980-8d70-40168d1c14fb-kube-api-access-sjjdr\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-slash\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-multus-socket-dir-parent\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.573071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-var-lib-kubelet\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecd20280-4ff2-42de-b4e2-85cdf88de891-serviceca\") pod \"node-ca-5k6jr\" (UID: \"ecd20280-4ff2-42de-b4e2-85cdf88de891\") " pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-multus-conf-dir\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-systemd\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1739a3c1-1f89-4206-b475-4c4b7374c36d-tmp\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-cnibin\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572639    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a3e4383-5637-45fd-b49d-9d77f7573feb-cni-binary-copy\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecd20280-4ff2-42de-b4e2-85cdf88de891-host\") pod \"node-ca-5k6jr\" (UID: \"ecd20280-4ff2-42de-b4e2-85cdf88de891\") " pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-hostroot\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d79a647-5223-4980-8d70-40168d1c14fb-multus-daemon-config\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-kubelet\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-run-openvswitch\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d4p6\" (UniqueName: \"kubernetes.io/projected/52951973-c105-489a-9d13-b213aff03d3d-kube-api-access-2d4p6\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-run-multus-certs\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.572963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhfl\" (UniqueName: \"kubernetes.io/projected/52ce0a05-4c82-4b03-8386-6ebdb0a6bae5-kube-api-access-mfhfl\") pod \"node-resolver-pspvw\" (UID: \"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5\") " pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.573829 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-system-cni-dir\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-multus-cni-dir\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-os-release\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573130    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19d542ed-18cc-46f9-bf7a-959f70dd7077-konnectivity-ca\") pod \"konnectivity-agent-mtz4k\" (UID: \"19d542ed-18cc-46f9-bf7a-959f70dd7077\") " pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-log-socket\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573174    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a3e4383-5637-45fd-b49d-9d77f7573feb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573221    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-os-release\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573297    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a3e4383-5637-45fd-b49d-9d77f7573feb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-lib-modules\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573388    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-run-netns\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-run-systemd\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-sysctl-d\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573436    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-etc-selinux\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb7bb954-4346-4175-8721-5ffbab5ef3d8-host-slash\") pod \"iptables-alerter-zk5nh\" (UID: \"bb7bb954-4346-4175-8721-5ffbab5ef3d8\") " pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-systemd-units\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.574463 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb7bb954-4346-4175-8721-5ffbab5ef3d8-iptables-alerter-script\") pod \"iptables-alerter-zk5nh\" (UID: \"bb7bb954-4346-4175-8721-5ffbab5ef3d8\") " pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-var-lib-kubelet\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-tuned\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-socket-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573567    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb7bb954-4346-4175-8721-5ffbab5ef3d8-host-slash\") pod \"iptables-alerter-zk5nh\" (UID: \"bb7bb954-4346-4175-8721-5ffbab5ef3d8\") " pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-system-cni-dir\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52951973-c105-489a-9d13-b213aff03d3d-env-overrides\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-kubernetes\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573648    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-system-cni-dir\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573706    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-socket-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-registration-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.573759    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573815    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-registration-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-cnibin\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573856    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a3e4383-5637-45fd-b49d-9d77f7573feb-cnibin\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.575034 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.573927    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret podName:6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:50.073893807 +0000 UTC m=+2.981774303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret") pod "global-pull-secret-syncer-2czf6" (UID: "6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:49.575637 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-etc-kubernetes\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.575637 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.573979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-etc-openvswitch\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.575637 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.574004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-run-ovn\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.575637 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.574027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52951973-c105-489a-9d13-b213aff03d3d-ovn-node-metrics-cert\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.575637 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.574053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-host\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.575637 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.574079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.575637 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.574104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:49.575637 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.574128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ce0a05-4c82-4b03-8386-6ebdb0a6bae5-tmp-dir\") pod \"node-resolver-pspvw\" (UID: \"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5\") " pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.575637 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.574165    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ddb948a-df3c-4979-8e17-c33ac7763f34-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.575968 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.575660    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19d542ed-18cc-46f9-bf7a-959f70dd7077-agent-certs\") pod \"konnectivity-agent-mtz4k\" (UID: \"19d542ed-18cc-46f9-bf7a-959f70dd7077\") " pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:47:49.579852 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.579832    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:49.579852 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.579854    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:49.580048 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.579867    2576 projected.go:196] Error preparing data for projected volume kube-api-access-qbxmb for pod openshift-network-diagnostics/network-check-target-wsmkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:49.580048 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.579936    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb podName:e586e4d3-72be-41a1-8b52-48301c1b36f4 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:50.079920013 +0000 UTC m=+2.987800504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qbxmb" (UniqueName: "kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb") pod "network-check-target-wsmkh" (UID: "e586e4d3-72be-41a1-8b52-48301c1b36f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:49.581669 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.581612    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chcvn\" (UniqueName: \"kubernetes.io/projected/3a3e4383-5637-45fd-b49d-9d77f7573feb-kube-api-access-chcvn\") pod \"multus-additional-cni-plugins-f2s9g\" (UID: \"3a3e4383-5637-45fd-b49d-9d77f7573feb\") " pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.581669 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.581652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4hcp\" (UniqueName: \"kubernetes.io/projected/ecd20280-4ff2-42de-b4e2-85cdf88de891-kube-api-access-m4hcp\") pod \"node-ca-5k6jr\" (UID: \"ecd20280-4ff2-42de-b4e2-85cdf88de891\") " pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.581816 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.581745    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7plm\" (UniqueName: \"kubernetes.io/projected/bb7bb954-4346-4175-8721-5ffbab5ef3d8-kube-api-access-g7plm\") pod \"iptables-alerter-zk5nh\" (UID: \"bb7bb954-4346-4175-8721-5ffbab5ef3d8\") " pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.581816 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.581799    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplgn\" (UniqueName: \"kubernetes.io/projected/5ddb948a-df3c-4979-8e17-c33ac7763f34-kube-api-access-gplgn\") pod \"aws-ebs-csi-driver-node-dcjkp\" (UID: \"5ddb948a-df3c-4979-8e17-c33ac7763f34\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.674878 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.674842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-multus-socket-dir-parent\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675051 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.674940    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-multus-socket-dir-parent\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675051 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.674955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-var-lib-kubelet\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675051 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-multus-conf-dir\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675051 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675016    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-var-lib-kubelet\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675051 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-systemd\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.675051 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1739a3c1-1f89-4206-b475-4c4b7374c36d-tmp\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675061    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-multus-conf-dir\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-cnibin\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-hostroot\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d79a647-5223-4980-8d70-40168d1c14fb-multus-daemon-config\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675139    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-hostroot\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675110    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-cnibin\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-systemd\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-kubelet\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-run-openvswitch\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d4p6\" (UniqueName: \"kubernetes.io/projected/52951973-c105-489a-9d13-b213aff03d3d-kube-api-access-2d4p6\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-run-multus-certs\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhfl\" (UniqueName: \"kubernetes.io/projected/52ce0a05-4c82-4b03-8386-6ebdb0a6bae5-kube-api-access-mfhfl\") pod \"node-resolver-pspvw\" (UID: \"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5\") " pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675361    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-run-openvswitch\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675339    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-kubelet\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.675381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675377    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-system-cni-dir\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675411    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-run-multus-certs\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-multus-cni-dir\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-os-release\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-multus-cni-dir\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-log-socket\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675505    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-system-cni-dir\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675553    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-os-release\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675587    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-log-socket\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-lib-modules\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675626    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-run-netns\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675664    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-run-netns\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-run-systemd\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-sysctl-d\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675694    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-run-systemd\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-lib-modules\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676052 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-systemd-units\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675737    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d79a647-5223-4980-8d70-40168d1c14fb-multus-daemon-config\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-var-lib-kubelet\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675746    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-systemd-units\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-tuned\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675804    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-sysctl-d\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52951973-c105-489a-9d13-b213aff03d3d-env-overrides\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675822    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-var-lib-kubelet\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-kubernetes\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.675976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-etc-kubernetes\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-etc-openvswitch\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-run-ovn\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52951973-c105-489a-9d13-b213aff03d3d-ovn-node-metrics-cert\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-host\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676081    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-kubernetes\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ce0a05-4c82-4b03-8386-6ebdb0a6bae5-tmp-dir\") pod \"node-resolver-pspvw\" (UID: \"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5\") " pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-run-k8s-cni-cncf-io\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676147    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-etc-kubernetes\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.676910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-cni-netd\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-run-ovn\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8q6j\" (UniqueName: \"kubernetes.io/projected/1739a3c1-1f89-4206-b475-4c4b7374c36d-kube-api-access-f8q6j\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676211    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-run-k8s-cni-cncf-io\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt8dv\" (UniqueName: \"kubernetes.io/projected/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-kube-api-access-vt8dv\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676241    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-host\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d79a647-5223-4980-8d70-40168d1c14fb-cni-binary-copy\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-modprobe-d\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676293    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-cni-netd\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-sysconfig\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-sys\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52ce0a05-4c82-4b03-8386-6ebdb0a6bae5-hosts-file\") pod \"node-resolver-pspvw\" (UID: \"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5\") " pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-var-lib-cni-multus\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-cni-bin\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52951973-c105-489a-9d13-b213aff03d3d-ovnkube-config\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676187    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-etc-openvswitch\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-var-lib-cni-bin\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-var-lib-openvswitch\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.677673 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676506    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ce0a05-4c82-4b03-8386-6ebdb0a6bae5-tmp-dir\") pod \"node-resolver-pspvw\" (UID: \"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5\") " pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676508    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-sys\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676550    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52951973-c105-489a-9d13-b213aff03d3d-env-overrides\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-sysctl-conf\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-cni-bin\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-run\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676639    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52ce0a05-4c82-4b03-8386-6ebdb0a6bae5-hosts-file\") pod \"node-resolver-pspvw\" (UID: \"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5\") " pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-run-netns\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676667    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-sysctl-conf\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-node-log\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52951973-c105-489a-9d13-b213aff03d3d-ovnkube-script-lib\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.676706    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjjdr\" (UniqueName: \"kubernetes.io/projected/8d79a647-5223-4980-8d70-40168d1c14fb-kube-api-access-sjjdr\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-slash\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:49.676758    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs podName:c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:50.176740907 +0000 UTC m=+3.084621429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs") pod "network-metrics-daemon-cfxz2" (UID: "c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:49.678327 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676575    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676787    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-slash\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676815    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-var-lib-cni-multus\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676817    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-run\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676837    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-host-run-netns\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676892    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-node-log\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.676932    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-modprobe-d\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.677055    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52951973-c105-489a-9d13-b213aff03d3d-ovnkube-config\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.677113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d79a647-5223-4980-8d70-40168d1c14fb-host-var-lib-cni-bin\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.677119    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52951973-c105-489a-9d13-b213aff03d3d-var-lib-openvswitch\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.677156    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d79a647-5223-4980-8d70-40168d1c14fb-cni-binary-copy\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.677173    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-sysconfig\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.677379    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52951973-c105-489a-9d13-b213aff03d3d-ovnkube-script-lib\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.678188    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1739a3c1-1f89-4206-b475-4c4b7374c36d-tmp\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.678999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.678798    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52951973-c105-489a-9d13-b213aff03d3d-ovn-node-metrics-cert\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.679474 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.679043    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1739a3c1-1f89-4206-b475-4c4b7374c36d-etc-tuned\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.684173 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.684072    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhfl\" (UniqueName: \"kubernetes.io/projected/52ce0a05-4c82-4b03-8386-6ebdb0a6bae5-kube-api-access-mfhfl\") pod \"node-resolver-pspvw\" (UID: \"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5\") " pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.684315 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.684289    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt8dv\" (UniqueName: \"kubernetes.io/projected/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-kube-api-access-vt8dv\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:49.685292 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.685228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjjdr\" (UniqueName: \"kubernetes.io/projected/8d79a647-5223-4980-8d70-40168d1c14fb-kube-api-access-sjjdr\") pod \"multus-xf8tc\" (UID: \"8d79a647-5223-4980-8d70-40168d1c14fb\") " pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.685535 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.685513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d4p6\" (UniqueName: \"kubernetes.io/projected/52951973-c105-489a-9d13-b213aff03d3d-kube-api-access-2d4p6\") pod \"ovnkube-node-gb7k9\" (UID: \"52951973-c105-489a-9d13-b213aff03d3d\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.688154 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.688134    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8q6j\" (UniqueName: \"kubernetes.io/projected/1739a3c1-1f89-4206-b475-4c4b7374c36d-kube-api-access-f8q6j\") pod \"tuned-sckf9\" (UID: \"1739a3c1-1f89-4206-b475-4c4b7374c36d\") " pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:49.766292 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.766235    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5k6jr"
Jun 15 05:47:49.772952 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.772929    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:47:49.779603 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.779582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f2s9g"
Jun 15 05:47:49.787008 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.786989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp"
Jun 15 05:47:49.795474 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.795448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zk5nh"
Jun 15 05:47:49.803928 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.803906    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pspvw"
Jun 15 05:47:49.809624 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.809605    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:47:49.816377 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.816351    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xf8tc"
Jun 15 05:47:49.821898 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:49.821836    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sckf9"
Jun 15 05:47:50.082926 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.080559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:50.082926 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.080732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:50.082926 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:50.080894    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:50.082926 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:50.080961    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret podName:6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:51.080944422 +0000 UTC m=+3.988824913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret") pod "global-pull-secret-syncer-2czf6" (UID: "6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:50.082926 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:50.081377    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:50.082926 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:50.081408    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:50.082926 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:50.081422    2576 projected.go:196] Error preparing data for projected volume kube-api-access-qbxmb for pod openshift-network-diagnostics/network-check-target-wsmkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:50.082926 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:50.081483    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb podName:e586e4d3-72be-41a1-8b52-48301c1b36f4 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:51.081467611 +0000 UTC m=+3.989348115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbxmb" (UniqueName: "kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb") pod "network-check-target-wsmkh" (UID: "e586e4d3-72be-41a1-8b52-48301c1b36f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:50.179370 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:50.179339    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d542ed_18cc_46f9_bf7a_959f70dd7077.slice/crio-a5656b712683d05b70fe22087b1b5367387f5aa93e9e35daf8ac49d4f69cb06d WatchSource:0}: Error finding container a5656b712683d05b70fe22087b1b5367387f5aa93e9e35daf8ac49d4f69cb06d: Status 404 returned error can't find the container with id a5656b712683d05b70fe22087b1b5367387f5aa93e9e35daf8ac49d4f69cb06d
Jun 15 05:47:50.180354 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:50.180316    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ce0a05_4c82_4b03_8386_6ebdb0a6bae5.slice/crio-d104d415b1c0526f2a5f7c40686cf01ffa6434c075ba1af84258c1b73a6c6e53 WatchSource:0}: Error finding container d104d415b1c0526f2a5f7c40686cf01ffa6434c075ba1af84258c1b73a6c6e53: Status 404 returned error can't find the container with id d104d415b1c0526f2a5f7c40686cf01ffa6434c075ba1af84258c1b73a6c6e53
Jun 15 05:47:50.181531 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.181316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:50.181531 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:50.181494    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:50.181686 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:50.181561    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs podName:c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:51.181544786 +0000 UTC m=+4.089425280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs") pod "network-metrics-daemon-cfxz2" (UID: "c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:50.182827 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:50.182807    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d79a647_5223_4980_8d70_40168d1c14fb.slice/crio-37f1d9413eea7f2d68079dd1c5be3ad4918184b7767bdf15dd33d06ce5c12287 WatchSource:0}: Error finding container 37f1d9413eea7f2d68079dd1c5be3ad4918184b7767bdf15dd33d06ce5c12287: Status 404 returned error can't find the container with id 37f1d9413eea7f2d68079dd1c5be3ad4918184b7767bdf15dd33d06ce5c12287
Jun 15 05:47:50.183639 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:50.183611    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7bb954_4346_4175_8721_5ffbab5ef3d8.slice/crio-444b2e088997eb5f33b9f1938a203dea70e0e0c01e089f7d8dea63371c7274d4 WatchSource:0}: Error finding container 444b2e088997eb5f33b9f1938a203dea70e0e0c01e089f7d8dea63371c7274d4: Status 404 returned error can't find the container with id 444b2e088997eb5f33b9f1938a203dea70e0e0c01e089f7d8dea63371c7274d4
Jun 15 05:47:50.185133 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:50.185081    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1739a3c1_1f89_4206_b475_4c4b7374c36d.slice/crio-b717bf76344eaebc2bd9022eaf528299cf0d24f8cc95127e4547b0a5c6638a6b WatchSource:0}: Error finding container b717bf76344eaebc2bd9022eaf528299cf0d24f8cc95127e4547b0a5c6638a6b: Status 404 returned error can't find the container with id b717bf76344eaebc2bd9022eaf528299cf0d24f8cc95127e4547b0a5c6638a6b
Jun 15 05:47:50.185798 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:50.185778    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd20280_4ff2_42de_b4e2_85cdf88de891.slice/crio-b2533c1b022a26a2628b934f41a4906ce9d078e573f35372a43e885168b6381c WatchSource:0}: Error finding container b2533c1b022a26a2628b934f41a4906ce9d078e573f35372a43e885168b6381c: Status 404 returned error can't find the container with id b2533c1b022a26a2628b934f41a4906ce9d078e573f35372a43e885168b6381c
Jun 15 05:47:50.187226 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:50.187055    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52951973_c105_489a_9d13_b213aff03d3d.slice/crio-668a5a1c206dc4285eb0b854d5b6d73c590438db388f302e51bc9f3b54e86425 WatchSource:0}: Error finding container 668a5a1c206dc4285eb0b854d5b6d73c590438db388f302e51bc9f3b54e86425: Status 404 returned error can't find the container with id 668a5a1c206dc4285eb0b854d5b6d73c590438db388f302e51bc9f3b54e86425
Jun 15 05:47:50.189578 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:50.189411    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3e4383_5637_45fd_b49d_9d77f7573feb.slice/crio-2661d4293496ce0973e75347a6ed3850267fb3431a721403f384fe41e8ff1008 WatchSource:0}: Error finding container 2661d4293496ce0973e75347a6ed3850267fb3431a721403f384fe41e8ff1008: Status 404 returned error can't find the container with id 2661d4293496ce0973e75347a6ed3850267fb3431a721403f384fe41e8ff1008
Jun 15 05:47:50.192672 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:47:50.192633    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddb948a_df3c_4979_8e17_c33ac7763f34.slice/crio-b178fe85f63890657e69f7f455cdf6c34c9cbc197b460bdb2e36121406a9db36 WatchSource:0}: Error finding container b178fe85f63890657e69f7f455cdf6c34c9cbc197b460bdb2e36121406a9db36: Status 404 returned error can't find the container with id b178fe85f63890657e69f7f455cdf6c34c9cbc197b460bdb2e36121406a9db36
Jun 15 05:47:50.506584 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.506490    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:42:49 +0000 UTC" deadline="2027-12-17 06:35:43.183820908 +0000 UTC"
Jun 15 05:47:50.506584 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.506547    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13200h47m52.677277699s"
Jun 15 05:47:50.546568 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.545794    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal" event={"ID":"cbc9805fb56ac37d1dbd6954f1bb4020","Type":"ContainerStarted","Data":"bc2092240be8faa0641a6f6e06c66c7914c207b9430434cc187a75d49dad4e99"}
Jun 15 05:47:50.548952 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.548925    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" event={"ID":"3a3e4383-5637-45fd-b49d-9d77f7573feb","Type":"ContainerStarted","Data":"2661d4293496ce0973e75347a6ed3850267fb3431a721403f384fe41e8ff1008"}
Jun 15 05:47:50.552117 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.552022    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerStarted","Data":"668a5a1c206dc4285eb0b854d5b6d73c590438db388f302e51bc9f3b54e86425"}
Jun 15 05:47:50.556429 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.556349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pspvw" event={"ID":"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5","Type":"ContainerStarted","Data":"d104d415b1c0526f2a5f7c40686cf01ffa6434c075ba1af84258c1b73a6c6e53"}
Jun 15 05:47:50.558089 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.558004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp" event={"ID":"5ddb948a-df3c-4979-8e17-c33ac7763f34","Type":"ContainerStarted","Data":"b178fe85f63890657e69f7f455cdf6c34c9cbc197b460bdb2e36121406a9db36"}
Jun 15 05:47:50.559796 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.559711    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5k6jr" event={"ID":"ecd20280-4ff2-42de-b4e2-85cdf88de891","Type":"ContainerStarted","Data":"b2533c1b022a26a2628b934f41a4906ce9d078e573f35372a43e885168b6381c"}
Jun 15 05:47:50.562569 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.562516    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-86.ec2.internal" podStartSLOduration=2.562498507 podStartE2EDuration="2.562498507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:50.561819817 +0000 UTC m=+3.469700331" watchObservedRunningTime="2026-06-15 05:47:50.562498507 +0000 UTC m=+3.470379021"
Jun 15 05:47:50.565888 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.565807    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sckf9" event={"ID":"1739a3c1-1f89-4206-b475-4c4b7374c36d","Type":"ContainerStarted","Data":"b717bf76344eaebc2bd9022eaf528299cf0d24f8cc95127e4547b0a5c6638a6b"}
Jun 15 05:47:50.571236 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.571206    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zk5nh" event={"ID":"bb7bb954-4346-4175-8721-5ffbab5ef3d8","Type":"ContainerStarted","Data":"444b2e088997eb5f33b9f1938a203dea70e0e0c01e089f7d8dea63371c7274d4"}
Jun 15 05:47:50.576747 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.576715    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xf8tc" event={"ID":"8d79a647-5223-4980-8d70-40168d1c14fb","Type":"ContainerStarted","Data":"37f1d9413eea7f2d68079dd1c5be3ad4918184b7767bdf15dd33d06ce5c12287"}
Jun 15 05:47:50.578449 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.578426    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mtz4k" event={"ID":"19d542ed-18cc-46f9-bf7a-959f70dd7077","Type":"ContainerStarted","Data":"a5656b712683d05b70fe22087b1b5367387f5aa93e9e35daf8ac49d4f69cb06d"}
Jun 15 05:47:50.582905 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.582879    2576 generic.go:356] "Generic (PLEG): container finished" podID="fc9ec5d46dd6686220f17504d67a7115" containerID="407a46cc4695a2e10d1be3fa8f78b80ea7e6eff81231125c514d09fb568a1f70" exitCode=0
Jun 15 05:47:50.583013 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:50.582915    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal" event={"ID":"fc9ec5d46dd6686220f17504d67a7115","Type":"ContainerDied","Data":"407a46cc4695a2e10d1be3fa8f78b80ea7e6eff81231125c514d09fb568a1f70"}
Jun 15 05:47:51.090960 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:51.090861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:51.090960 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:51.090921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:51.091194 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.091090    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:51.091194 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.091110    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:51.091194 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.091123    2576 projected.go:196] Error preparing data for projected volume kube-api-access-qbxmb for pod openshift-network-diagnostics/network-check-target-wsmkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:51.091194 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.091178    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb podName:e586e4d3-72be-41a1-8b52-48301c1b36f4 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:53.091160243 +0000 UTC m=+5.999040742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbxmb" (UniqueName: "kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb") pod "network-check-target-wsmkh" (UID: "e586e4d3-72be-41a1-8b52-48301c1b36f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:51.091655 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.091635    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:51.091725 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.091698    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret podName:6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:53.091683153 +0000 UTC m=+5.999563649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret") pod "global-pull-secret-syncer-2czf6" (UID: "6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:51.191607 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:51.191561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:51.191814 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.191712    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:51.191814 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.191776    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs podName:c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:53.191757009 +0000 UTC m=+6.099637501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs") pod "network-metrics-daemon-cfxz2" (UID: "c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:51.506889 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:51.506799    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:42:49 +0000 UTC" deadline="2028-02-14 18:13:09.205111977 +0000 UTC"
Jun 15 05:47:51.506889 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:51.506840    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14628h25m17.69827616s"
Jun 15 05:47:51.533577 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:51.533538    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:51.534004 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.533967    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:47:51.534738 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:51.533599    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:51.534738 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.534567    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:47:51.534969 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:51.533651    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:51.535135 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:51.535095    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:47:51.608094 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:51.608002    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal" event={"ID":"fc9ec5d46dd6686220f17504d67a7115","Type":"ContainerStarted","Data":"b4f0f580ac08a2cefb39136d4fc43cf8d82730fd11805df5a032e8da5ede4e24"}
Jun 15 05:47:53.109584 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:53.109521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:53.109584 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:53.109577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:53.110105 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.109755    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:53.110105 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.109776    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:53.110105 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.109791    2576 projected.go:196] Error preparing data for projected volume kube-api-access-qbxmb for pod openshift-network-diagnostics/network-check-target-wsmkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:53.110105 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.109852    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb podName:e586e4d3-72be-41a1-8b52-48301c1b36f4 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:57.109832246 +0000 UTC m=+10.017712739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbxmb" (UniqueName: "kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb") pod "network-check-target-wsmkh" (UID: "e586e4d3-72be-41a1-8b52-48301c1b36f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:53.110344 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.110281    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:53.110344 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.110336    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret podName:6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:57.110320223 +0000 UTC m=+10.018200749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret") pod "global-pull-secret-syncer-2czf6" (UID: "6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:53.210389 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:53.210347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:53.210583 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.210523    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:53.210652 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.210592    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs podName:c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:57.210572371 +0000 UTC m=+10.118452867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs") pod "network-metrics-daemon-cfxz2" (UID: "c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:53.529424 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:53.528358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:53.529424 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.528491    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:47:53.529424 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:53.529162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:53.529424 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.529272    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:47:53.530223 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:53.529801    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:53.530223 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:53.529918    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:47:55.528240 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:55.528205    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:55.528707 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:55.528397    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:47:55.528707 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:55.528451    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:55.528707 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:55.528549    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:47:55.528707 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:55.528600    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:55.528707 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:55.528667    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:47:57.144545 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:57.144500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:57.144984 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:57.144563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:57.144984 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.144738    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:57.144984 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.144758    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:57.144984 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.144771    2576 projected.go:196] Error preparing data for projected volume kube-api-access-qbxmb for pod openshift-network-diagnostics/network-check-target-wsmkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:57.144984 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.144837    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb podName:e586e4d3-72be-41a1-8b52-48301c1b36f4 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:05.144818326 +0000 UTC m=+18.052698831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbxmb" (UniqueName: "kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb") pod "network-check-target-wsmkh" (UID: "e586e4d3-72be-41a1-8b52-48301c1b36f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:57.145393 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.145292    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:57.145393 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.145356    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret podName:6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:05.145338572 +0000 UTC m=+18.053219067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret") pod "global-pull-secret-syncer-2czf6" (UID: "6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:57.245487 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:57.245448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:57.245673 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.245614    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:57.245734 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.245693    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs podName:c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:05.245671384 +0000 UTC m=+18.153551880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs") pod "network-metrics-daemon-cfxz2" (UID: "c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:57.528160 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:57.528085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:57.528343 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.528165    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:47:57.528546 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:57.528528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:57.528675 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.528635    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:47:57.528780 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:57.528766    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:57.528871 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:57.528847    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:47:59.527595 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:59.527556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:47:59.528023 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:59.527556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:47:59.528023 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:59.527711    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:47:59.528023 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:59.527801    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:47:59.528023 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:47:59.527556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:47:59.528023 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:47:59.527880    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:01.528086 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:01.528047    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:01.528577 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:01.528047    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:01.528577 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:01.528189    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:01.528577 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:01.528203    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:01.528577 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:01.528314    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:01.528577 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:01.528412    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:03.527754 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:03.527709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:03.528215 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:03.527835    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:03.528215 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:03.527845    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:03.528215 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:03.527965    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:03.528215 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:03.528013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:03.528215 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:03.528088    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:05.211495 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:05.211452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:05.211495 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:05.211503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:05.212153 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.211619    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:05.212153 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.211667    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:05.212153 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.211683    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:05.212153 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.211686    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret podName:6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:21.211668473 +0000 UTC m=+34.119548968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret") pod "global-pull-secret-syncer-2czf6" (UID: "6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:05.212153 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.211693    2576 projected.go:196] Error preparing data for projected volume kube-api-access-qbxmb for pod openshift-network-diagnostics/network-check-target-wsmkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:05.212153 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.211732    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb podName:e586e4d3-72be-41a1-8b52-48301c1b36f4 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:21.211722527 +0000 UTC m=+34.119603033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbxmb" (UniqueName: "kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb") pod "network-check-target-wsmkh" (UID: "e586e4d3-72be-41a1-8b52-48301c1b36f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:05.312370 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:05.312328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:05.312528 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.312490    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:05.312583 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.312568    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs podName:c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:21.312548914 +0000 UTC m=+34.220429426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs") pod "network-metrics-daemon-cfxz2" (UID: "c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:05.528207 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:05.528119    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:05.528207 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:05.528137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:05.528440 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:05.528126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:05.528440 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.528264    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:05.528440 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.528342    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:05.528440 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:05.528428    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:07.528121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:07.528087    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:07.528593 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:07.528196    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:07.528593 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:07.528284    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:07.528593 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:07.528364    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:07.528593 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:07.528406    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:07.528593 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:07.528481    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:09.528135 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.527953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:09.528662 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.527956    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:09.528662 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:09.528267    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:09.528662 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:09.528392    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:09.528662 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.527998    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:09.528662 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:09.528488    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:09.656071 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.656043    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a3e4383-5637-45fd-b49d-9d77f7573feb" containerID="4e51f8d00e120d1e53aa3f68cd39ce7005ae40e58dfb3b21ae92d5e6a61d2e44" exitCode=0
Jun 15 05:48:09.656197 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.656118    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" event={"ID":"3a3e4383-5637-45fd-b49d-9d77f7573feb","Type":"ContainerDied","Data":"4e51f8d00e120d1e53aa3f68cd39ce7005ae40e58dfb3b21ae92d5e6a61d2e44"}
Jun 15 05:48:09.658446 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.658428    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 05:48:09.658806 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.658782    2576 generic.go:356] "Generic (PLEG): container finished" podID="52951973-c105-489a-9d13-b213aff03d3d" containerID="45e3fc1261d258faa9e6be63d79802dc2c061c8cd74eae4be2511abbd67041bb" exitCode=1
Jun 15 05:48:09.658894 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.658851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerStarted","Data":"ae02a418ba74e0f22c5374ab53aed0936701066a0a2507e009ef6085965ce3e2"}
Jun 15 05:48:09.658954 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.658920    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerStarted","Data":"1bc8f5647222ff1f81d10c312853e91d2b8311e0caa26ee59aa952399587ce82"}
Jun 15 05:48:09.659059 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.658968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerDied","Data":"45e3fc1261d258faa9e6be63d79802dc2c061c8cd74eae4be2511abbd67041bb"}
Jun 15 05:48:09.659059 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.659001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerStarted","Data":"79d8e8234c4d2c8564069e53019ba0e7458d35a70bee9348ab22b27d8568d845"}
Jun 15 05:48:09.660370 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.660349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pspvw" event={"ID":"52ce0a05-4c82-4b03-8386-6ebdb0a6bae5","Type":"ContainerStarted","Data":"2d3d1ddc84f45c6456abc34e5ac4e92cae3944762b81ac764f18db13c59623be"}
Jun 15 05:48:09.661889 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.661865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp" event={"ID":"5ddb948a-df3c-4979-8e17-c33ac7763f34","Type":"ContainerStarted","Data":"b78d975f36a86b20130c54d43f50d2c40c108f6283c74c22008dbbafcdbe9fe2"}
Jun 15 05:48:09.663209 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.663190    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5k6jr" event={"ID":"ecd20280-4ff2-42de-b4e2-85cdf88de891","Type":"ContainerStarted","Data":"8367fade1c3e81ad7714a7fbe81afb2f8bdd6b27dcbe11c10117b0e949163afd"}
Jun 15 05:48:09.664619 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.664601    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sckf9" event={"ID":"1739a3c1-1f89-4206-b475-4c4b7374c36d","Type":"ContainerStarted","Data":"fc5de910403ba37e88126040e396f670f7cab84202416e0337fd2497561a9ff8"}
Jun 15 05:48:09.665934 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.665915    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xf8tc" event={"ID":"8d79a647-5223-4980-8d70-40168d1c14fb","Type":"ContainerStarted","Data":"369b994787da0d3ed53ea769e40ef5b624437029c9960bc6c0636e08d2e69e7b"}
Jun 15 05:48:09.667103 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.667085    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mtz4k" event={"ID":"19d542ed-18cc-46f9-bf7a-959f70dd7077","Type":"ContainerStarted","Data":"7cb9ae50032fc1744b9eebb8ce8c04596ecacda889fdd65d186d306425cc641c"}
Jun 15 05:48:09.685470 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.685421    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-86.ec2.internal" podStartSLOduration=21.685405737 podStartE2EDuration="21.685405737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:51.626720443 +0000 UTC m=+4.534600957" watchObservedRunningTime="2026-06-15 05:48:09.685405737 +0000 UTC m=+22.593286250"
Jun 15 05:48:09.699991 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.699941    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xf8tc" podStartSLOduration=3.5893641609999998 podStartE2EDuration="22.69992096s" totalImagesPullingTime="19.110556799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:09.699435584 +0000 UTC m=+22.607316097" watchObservedRunningTime="2026-06-15 05:48:09.69992096 +0000 UTC m=+22.607801510"
Jun 15 05:48:09.717152 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.717052    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sckf9" podStartSLOduration=3.724778385 podStartE2EDuration="22.717031798s" totalImagesPullingTime="18.992253413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:09.716404679 +0000 UTC m=+22.624285193" watchObservedRunningTime="2026-06-15 05:48:09.717031798 +0000 UTC m=+22.624912309"
Jun 15 05:48:09.730984 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.730928    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5k6jr" podStartSLOduration=3.737328759 podStartE2EDuration="22.730908783s" totalImagesPullingTime="18.993580024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:09.730161481 +0000 UTC m=+22.638042007" watchObservedRunningTime="2026-06-15 05:48:09.730908783 +0000 UTC m=+22.638789296"
Jun 15 05:48:09.744861 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:09.744809    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mtz4k" podStartSLOduration=3.775962167 podStartE2EDuration="22.744788542s" totalImagesPullingTime="18.968826375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:09.744342418 +0000 UTC m=+22.652222932" watchObservedRunningTime="2026-06-15 05:48:09.744788542 +0000 UTC m=+22.652669057"
Jun 15 05:48:10.442037 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:10.442006    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 15 05:48:10.528714 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:10.528331    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-15T05:48:10.442032461Z","UUID":"2e446441-0fb1-4e3d-978b-18ec6f596ccf","Handler":null,"Name":"","Endpoint":""}
Jun 15 05:48:10.529911 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:10.529868    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 15 05:48:10.530070 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:10.530055    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 15 05:48:10.673383 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:10.673352    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 05:48:10.673854 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:10.673824    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerStarted","Data":"f4bcc6e9503734b62d874f621119f80ee1d3ca988a9a5a1dfda9cdeebf822f32"}
Jun 15 05:48:10.673981 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:10.673962    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerStarted","Data":"3b6fe981dd71b5ff44a9432680215c51961f1005a5acc498e9861ab291b03e23"}
Jun 15 05:48:10.675809 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:10.675776    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp" event={"ID":"5ddb948a-df3c-4979-8e17-c33ac7763f34","Type":"ContainerStarted","Data":"9a4a44968c31e3e306324492bf214b62e76ad3070c6a2c6f00bdf307339129f7"}
Jun 15 05:48:11.462896 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.462855    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:48:11.463467 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.463430    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:48:11.480715 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.480664    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pspvw" podStartSLOduration=5.47561534 podStartE2EDuration="24.480647653s" totalImagesPullingTime="19.005032313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:09.758911644 +0000 UTC m=+22.666792171" watchObservedRunningTime="2026-06-15 05:48:11.480647653 +0000 UTC m=+24.388528225"
Jun 15 05:48:11.527771 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.527732    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:11.527937 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.527733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:11.527937 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.527902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:11.527937 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:11.527921    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:11.528090 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:11.528010    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:11.529545 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:11.528462    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:11.680638 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.680590    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp" event={"ID":"5ddb948a-df3c-4979-8e17-c33ac7763f34","Type":"ContainerStarted","Data":"33cef9ca4c9a77d5d9a4af08ca9cd6e9b22b59a47e6bc6f3e51934ed24542133"}
Jun 15 05:48:11.682389 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.682319    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zk5nh" event={"ID":"bb7bb954-4346-4175-8721-5ffbab5ef3d8","Type":"ContainerStarted","Data":"44fc089afa2a3e53ad023b3d61431f855e7f80bc28e22802423ec8380593cb68"}
Jun 15 05:48:11.682769 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.682574    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:48:11.682911 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.682895    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mtz4k"
Jun 15 05:48:11.701747 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.701690    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dcjkp" podStartSLOduration=3.694965742 podStartE2EDuration="24.70167087s" totalImagesPullingTime="21.006705128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:11.701070533 +0000 UTC m=+24.608951045" watchObservedRunningTime="2026-06-15 05:48:11.70167087 +0000 UTC m=+24.609551386"
Jun 15 05:48:11.733172 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:11.733066    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zk5nh" podStartSLOduration=5.7682137749999995 podStartE2EDuration="24.733043933s" totalImagesPullingTime="18.964830158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:11.717317087 +0000 UTC m=+24.625197598" watchObservedRunningTime="2026-06-15 05:48:11.733043933 +0000 UTC m=+24.640924448"
Jun 15 05:48:12.688768 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:12.688742    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 05:48:12.689410 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:12.689199    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerStarted","Data":"c53c2c37be3eb17d0771ad7029712f56e9eb90a8d5cd20f0d15a6393159081a2"}
Jun 15 05:48:13.527337 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:13.527299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:13.527337 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:13.527299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:13.527545 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:13.527355    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:13.527545 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:13.527465    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:13.527611 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:13.527576    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:13.527699 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:13.527681    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:14.697447 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:14.697282    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 05:48:14.698152 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:14.697833    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerStarted","Data":"1059a55d29a066503ba359ad4a47780d3c15f4d66e57d096e1cac9733098650f"}
Jun 15 05:48:14.698152 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:14.698045    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:48:14.698152 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:14.698099    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:48:14.698330 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:14.698313    2576 scope.go:117] "RemoveContainer" containerID="45e3fc1261d258faa9e6be63d79802dc2c061c8cd74eae4be2511abbd67041bb"
Jun 15 05:48:14.699919 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:14.699897    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a3e4383-5637-45fd-b49d-9d77f7573feb" containerID="213ab1781371115e9806590cf63024350050f29228c800ee4abdcf2b45751dd6" exitCode=0
Jun 15 05:48:14.699919 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:14.699926    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" event={"ID":"3a3e4383-5637-45fd-b49d-9d77f7573feb","Type":"ContainerDied","Data":"213ab1781371115e9806590cf63024350050f29228c800ee4abdcf2b45751dd6"}
Jun 15 05:48:14.715045 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:14.715017    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:48:15.527715 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:15.527680    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:15.527897 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:15.527686    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:15.527897 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:15.527810    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:15.527897 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:15.527809    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:15.528008 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:15.527927    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:15.528049 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:15.528008    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:15.705647 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:15.705619    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 05:48:15.706172 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:15.706062    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" event={"ID":"52951973-c105-489a-9d13-b213aff03d3d","Type":"ContainerStarted","Data":"d2e163aa1079db2a90073f123ceb3d3287d68fb9f719bda2249dd33cafabf0c2"}
Jun 15 05:48:15.706321 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:15.706289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:48:15.723097 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:15.723068    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:48:15.739826 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:15.739780    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9" podStartSLOduration=9.716496583 podStartE2EDuration="28.739766893s" totalImagesPullingTime="19.02327031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:15.738465974 +0000 UTC m=+28.646346488" watchObservedRunningTime="2026-06-15 05:48:15.739766893 +0000 UTC m=+28.647647407"
Jun 15 05:48:16.074262 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:16.073974    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wsmkh"]
Jun 15 05:48:16.074837 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:16.074329    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:16.074837 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:16.074436    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:16.075215 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:16.075190    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2czf6"]
Jun 15 05:48:16.075376 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:16.075318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:16.075432 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:16.075410    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:16.075860 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:16.075839    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cfxz2"]
Jun 15 05:48:16.075961 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:16.075947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:16.076038 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:16.076023    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:16.721970 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:16.721935    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a3e4383-5637-45fd-b49d-9d77f7573feb" containerID="9e866d1df747eaba3a89ac56e3efc0c148726e205c43a48055dc31dc74dab139" exitCode=0
Jun 15 05:48:16.722443 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:16.722018    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" event={"ID":"3a3e4383-5637-45fd-b49d-9d77f7573feb","Type":"ContainerDied","Data":"9e866d1df747eaba3a89ac56e3efc0c148726e205c43a48055dc31dc74dab139"}
Jun 15 05:48:17.528354 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:17.528321    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:17.528514 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:17.528435    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:17.528560 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:17.528515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:17.528652 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:17.528630    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:17.528693 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:17.528666    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:17.528760 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:17.528743    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:18.730794 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:18.730761    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a3e4383-5637-45fd-b49d-9d77f7573feb" containerID="aaacfaf2456625032e03e1625eac2dd69fc15f0524a4016ea56bf1d5aa37da43" exitCode=0
Jun 15 05:48:18.731366 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:18.730823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" event={"ID":"3a3e4383-5637-45fd-b49d-9d77f7573feb","Type":"ContainerDied","Data":"aaacfaf2456625032e03e1625eac2dd69fc15f0524a4016ea56bf1d5aa37da43"}
Jun 15 05:48:19.528138 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:19.528098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:19.528351 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:19.528105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:19.528351 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:19.528259    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:19.528351 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:19.528237    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:19.528517 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:19.528371    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:19.528517 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:19.528457    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:21.236189 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.236144    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:21.236650 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.236204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:21.236650 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.236327    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:21.236650 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.236340    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:21.236650 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.236355    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:21.236650 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.236367    2576 projected.go:196] Error preparing data for projected volume kube-api-access-qbxmb for pod openshift-network-diagnostics/network-check-target-wsmkh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:21.236650 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.236399    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret podName:6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:53.236379396 +0000 UTC m=+66.144259907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret") pod "global-pull-secret-syncer-2czf6" (UID: "6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:21.236650 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.236417    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb podName:e586e4d3-72be-41a1-8b52-48301c1b36f4 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:53.236407655 +0000 UTC m=+66.144288154 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbxmb" (UniqueName: "kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb") pod "network-check-target-wsmkh" (UID: "e586e4d3-72be-41a1-8b52-48301c1b36f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:21.337361 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.337322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:21.337538 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.337478    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:21.337590 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.337546    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs podName:c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:53.337526324 +0000 UTC m=+66.245406840 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs") pod "network-metrics-daemon-cfxz2" (UID: "c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:21.527797 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.527700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:21.527797 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.527703    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:21.527985 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.527839    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:21.527985 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.527836    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2czf6" podUID="6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53"
Jun 15 05:48:21.527985 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.527961    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsmkh" podUID="e586e4d3-72be-41a1-8b52-48301c1b36f4"
Jun 15 05:48:21.528097 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:21.528060    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:48:21.906696 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.906618    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-86.ec2.internal" event="NodeReady"
Jun 15 05:48:21.906856 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.906772    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jun 15 05:48:21.957242 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.957202    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kn75r"]
Jun 15 05:48:21.959590 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.959560    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hslrg"]
Jun 15 05:48:21.959745 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.959722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:21.961363 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.961340    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hslrg"
Jun 15 05:48:21.962605 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.962350    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5mfr5\""
Jun 15 05:48:21.962605 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.962415    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 15 05:48:21.962815 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.962794    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 15 05:48:21.963789 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.963619    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 15 05:48:21.963789 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.963679    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 15 05:48:21.963789 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.963689    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 15 05:48:21.963789 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.963726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8pttw\""
Jun 15 05:48:21.971059 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.971033    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hslrg"]
Jun 15 05:48:21.971896 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:21.971876    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kn75r"]
Jun 15 05:48:22.042714 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.042646    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbls\" (UniqueName: \"kubernetes.io/projected/029b7613-6544-4c5b-aef8-3041d968a75f-kube-api-access-wwbls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.042714 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.042708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029b7613-6544-4c5b-aef8-3041d968a75f-tmp-dir\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.042953 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.042743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/029b7613-6544-4c5b-aef8-3041d968a75f-config-volume\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.042953 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.042766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.143341 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.143305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3e29ab-7a28-4dd1-8d55-a3115197a16c-cert\") pod \"ingress-canary-hslrg\" (UID: \"1e3e29ab-7a28-4dd1-8d55-a3115197a16c\") " pod="openshift-ingress-canary/ingress-canary-hslrg"
Jun 15 05:48:22.143341 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.143357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbls\" (UniqueName: \"kubernetes.io/projected/029b7613-6544-4c5b-aef8-3041d968a75f-kube-api-access-wwbls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.143341 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.143386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029b7613-6544-4c5b-aef8-3041d968a75f-tmp-dir\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.143653 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.143464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/029b7613-6544-4c5b-aef8-3041d968a75f-config-volume\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.143653 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.143597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.143653 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.143644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmnr\" (UniqueName: \"kubernetes.io/projected/1e3e29ab-7a28-4dd1-8d55-a3115197a16c-kube-api-access-dvmnr\") pod \"ingress-canary-hslrg\" (UID: \"1e3e29ab-7a28-4dd1-8d55-a3115197a16c\") " pod="openshift-ingress-canary/ingress-canary-hslrg"
Jun 15 05:48:22.143756 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.143732    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029b7613-6544-4c5b-aef8-3041d968a75f-tmp-dir\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.143799 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:22.143771    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:22.143854 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:22.143839    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls podName:029b7613-6544-4c5b-aef8-3041d968a75f nodeName:}" failed. No retries permitted until 2026-06-15 05:48:22.643818568 +0000 UTC m=+35.551699080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls") pod "dns-default-kn75r" (UID: "029b7613-6544-4c5b-aef8-3041d968a75f") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:22.144070 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.144051    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/029b7613-6544-4c5b-aef8-3041d968a75f-config-volume\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.156002 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.155976    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbls\" (UniqueName: \"kubernetes.io/projected/029b7613-6544-4c5b-aef8-3041d968a75f-kube-api-access-wwbls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.244653 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.244611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmnr\" (UniqueName: \"kubernetes.io/projected/1e3e29ab-7a28-4dd1-8d55-a3115197a16c-kube-api-access-dvmnr\") pod \"ingress-canary-hslrg\" (UID: \"1e3e29ab-7a28-4dd1-8d55-a3115197a16c\") " pod="openshift-ingress-canary/ingress-canary-hslrg"
Jun 15 05:48:22.245105 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.244722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3e29ab-7a28-4dd1-8d55-a3115197a16c-cert\") pod \"ingress-canary-hslrg\" (UID: \"1e3e29ab-7a28-4dd1-8d55-a3115197a16c\") " pod="openshift-ingress-canary/ingress-canary-hslrg"
Jun 15 05:48:22.247318 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.247296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3e29ab-7a28-4dd1-8d55-a3115197a16c-cert\") pod \"ingress-canary-hslrg\" (UID: \"1e3e29ab-7a28-4dd1-8d55-a3115197a16c\") " pod="openshift-ingress-canary/ingress-canary-hslrg"
Jun 15 05:48:22.253790 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.253770    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmnr\" (UniqueName: \"kubernetes.io/projected/1e3e29ab-7a28-4dd1-8d55-a3115197a16c-kube-api-access-dvmnr\") pod \"ingress-canary-hslrg\" (UID: \"1e3e29ab-7a28-4dd1-8d55-a3115197a16c\") " pod="openshift-ingress-canary/ingress-canary-hslrg"
Jun 15 05:48:22.281762 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.281724    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hslrg"
Jun 15 05:48:22.648522 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.648493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:22.648677 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:22.648659    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:22.648736 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:22.648725    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls podName:029b7613-6544-4c5b-aef8-3041d968a75f nodeName:}" failed. No retries permitted until 2026-06-15 05:48:23.648709579 +0000 UTC m=+36.556590069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls") pod "dns-default-kn75r" (UID: "029b7613-6544-4c5b-aef8-3041d968a75f") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:22.681031 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.681005    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hslrg"]
Jun 15 05:48:22.684556 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:48:22.684530    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3e29ab_7a28_4dd1_8d55_a3115197a16c.slice/crio-d8c67e0f16ecd53fd267e76b92996c60f8c738a7893124b08dd45a7edc61cff7 WatchSource:0}: Error finding container d8c67e0f16ecd53fd267e76b92996c60f8c738a7893124b08dd45a7edc61cff7: Status 404 returned error can't find the container with id d8c67e0f16ecd53fd267e76b92996c60f8c738a7893124b08dd45a7edc61cff7
Jun 15 05:48:22.742841 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.742804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hslrg" event={"ID":"1e3e29ab-7a28-4dd1-8d55-a3115197a16c","Type":"ContainerStarted","Data":"d8c67e0f16ecd53fd267e76b92996c60f8c738a7893124b08dd45a7edc61cff7"}
Jun 15 05:48:22.745866 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:22.745823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" event={"ID":"3a3e4383-5637-45fd-b49d-9d77f7573feb","Type":"ContainerStarted","Data":"2d7667d4205b72d35a65cd36b745cf84474a8a1586d027f7ae9a23ec55640ae6"}
Jun 15 05:48:23.527671 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.527630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:23.528099 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.527630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:23.528099 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.527639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:23.530684 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.530661    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 15 05:48:23.530813 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.530765    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 15 05:48:23.530906 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.530889    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 15 05:48:23.531005 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.530978    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 15 05:48:23.531783 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.531735    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8qp7w\""
Jun 15 05:48:23.531901 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.531802    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s8jdr\""
Jun 15 05:48:23.657498 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.657301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:23.657680 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:23.657465    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:23.657680 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:23.657645    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls podName:029b7613-6544-4c5b-aef8-3041d968a75f nodeName:}" failed. No retries permitted until 2026-06-15 05:48:25.657602831 +0000 UTC m=+38.565483321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls") pod "dns-default-kn75r" (UID: "029b7613-6544-4c5b-aef8-3041d968a75f") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:23.751881 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.751845    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a3e4383-5637-45fd-b49d-9d77f7573feb" containerID="2d7667d4205b72d35a65cd36b745cf84474a8a1586d027f7ae9a23ec55640ae6" exitCode=0
Jun 15 05:48:23.752056 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:23.751907    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" event={"ID":"3a3e4383-5637-45fd-b49d-9d77f7573feb","Type":"ContainerDied","Data":"2d7667d4205b72d35a65cd36b745cf84474a8a1586d027f7ae9a23ec55640ae6"}
Jun 15 05:48:24.755969 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:24.755926    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hslrg" event={"ID":"1e3e29ab-7a28-4dd1-8d55-a3115197a16c","Type":"ContainerStarted","Data":"2832eae4491c0ecc25f67c4115fadbc6632b0de5ccce5a869e5e06f31262530a"}
Jun 15 05:48:24.758942 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:24.758914    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a3e4383-5637-45fd-b49d-9d77f7573feb" containerID="8750c938d7ed821fa6487924f25e6305a2bb552617e9ca1b1d32ab80ff781287" exitCode=0
Jun 15 05:48:24.759041 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:24.758970    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" event={"ID":"3a3e4383-5637-45fd-b49d-9d77f7573feb","Type":"ContainerDied","Data":"8750c938d7ed821fa6487924f25e6305a2bb552617e9ca1b1d32ab80ff781287"}
Jun 15 05:48:24.782526 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:24.782482    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hslrg" podStartSLOduration=2.325808092 podStartE2EDuration="3.78246809s" totalImagesPullingTime="1.456659998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:24.782441978 +0000 UTC m=+37.690322491" watchObservedRunningTime="2026-06-15 05:48:24.78246809 +0000 UTC m=+37.690348603"
Jun 15 05:48:25.670943 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:25.670908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:25.671105 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:25.671073    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:25.671154 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:25.671143    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls podName:029b7613-6544-4c5b-aef8-3041d968a75f nodeName:}" failed. No retries permitted until 2026-06-15 05:48:29.671126404 +0000 UTC m=+42.579006894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls") pod "dns-default-kn75r" (UID: "029b7613-6544-4c5b-aef8-3041d968a75f") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:25.764986 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:25.764948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" event={"ID":"3a3e4383-5637-45fd-b49d-9d77f7573feb","Type":"ContainerStarted","Data":"b502f93e296f056d6961170e81c06f7e6b06f02b716a541d700ef89d53fe7674"}
Jun 15 05:48:25.790712 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:25.790662    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f2s9g" podStartSLOduration=9.471742252 podStartE2EDuration="38.790645358s" totalImagesPullingTime="29.318903106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:25.789324187 +0000 UTC m=+38.697204701" watchObservedRunningTime="2026-06-15 05:48:25.790645358 +0000 UTC m=+38.698525919"
Jun 15 05:48:29.700415 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:29.700374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:29.700912 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:29.700556    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:29.700912 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:29.700643    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls podName:029b7613-6544-4c5b-aef8-3041d968a75f nodeName:}" failed. No retries permitted until 2026-06-15 05:48:37.700621544 +0000 UTC m=+50.608502038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls") pod "dns-default-kn75r" (UID: "029b7613-6544-4c5b-aef8-3041d968a75f") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:37.757703 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:37.757668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:37.758237 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:37.757842    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:37.758237 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:37.757912    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls podName:029b7613-6544-4c5b-aef8-3041d968a75f nodeName:}" failed. No retries permitted until 2026-06-15 05:48:53.757897616 +0000 UTC m=+66.665778110 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls") pod "dns-default-kn75r" (UID: "029b7613-6544-4c5b-aef8-3041d968a75f") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:47.742656 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:47.742619    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb7k9"
Jun 15 05:48:53.261863 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.261813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:53.262368 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.261912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:53.264594 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.264573    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 15 05:48:53.264594 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.264588    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 15 05:48:53.274749 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.274723    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 15 05:48:53.275607 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.275580    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53-original-pull-secret\") pod \"global-pull-secret-syncer-2czf6\" (UID: \"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53\") " pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:53.285053 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.285023    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxmb\" (UniqueName: \"kubernetes.io/projected/e586e4d3-72be-41a1-8b52-48301c1b36f4-kube-api-access-qbxmb\") pod \"network-check-target-wsmkh\" (UID: \"e586e4d3-72be-41a1-8b52-48301c1b36f4\") " pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:53.363114 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.363076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:48:53.365721 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.365700    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 15 05:48:53.373872 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:53.373843    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 15 05:48:53.373997 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:53.373923    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs podName:c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:57.373900739 +0000 UTC m=+130.281781231 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs") pod "network-metrics-daemon-cfxz2" (UID: "c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2") : secret "metrics-daemon-secret" not found
Jun 15 05:48:53.541269 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.541176    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s8jdr\""
Jun 15 05:48:53.545613 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.545598    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2czf6"
Jun 15 05:48:53.548814 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.548794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:53.733225 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.733045    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2czf6"]
Jun 15 05:48:53.736461 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:48:53.736434    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0fdb7a_b248_4f55_80eb_f9cc18e0ec53.slice/crio-ef8e0789e984565fdf5233a888c591dc16a7841d479813b56a86e2145a239048 WatchSource:0}: Error finding container ef8e0789e984565fdf5233a888c591dc16a7841d479813b56a86e2145a239048: Status 404 returned error can't find the container with id ef8e0789e984565fdf5233a888c591dc16a7841d479813b56a86e2145a239048
Jun 15 05:48:53.758786 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.758757    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wsmkh"]
Jun 15 05:48:53.761316 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:48:53.761290    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode586e4d3_72be_41a1_8b52_48301c1b36f4.slice/crio-d1f63866ac10fee4ff0c086c1c8a58c20e3170a977e0d47174b8d760d839fee2 WatchSource:0}: Error finding container d1f63866ac10fee4ff0c086c1c8a58c20e3170a977e0d47174b8d760d839fee2: Status 404 returned error can't find the container with id d1f63866ac10fee4ff0c086c1c8a58c20e3170a977e0d47174b8d760d839fee2
Jun 15 05:48:53.765797 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.765778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:48:53.765942 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:53.765924    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:53.765993 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:48:53.765982    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls podName:029b7613-6544-4c5b-aef8-3041d968a75f nodeName:}" failed. No retries permitted until 2026-06-15 05:49:25.765966499 +0000 UTC m=+98.673846990 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls") pod "dns-default-kn75r" (UID: "029b7613-6544-4c5b-aef8-3041d968a75f") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:53.832472 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.832379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2czf6" event={"ID":"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53","Type":"ContainerStarted","Data":"ef8e0789e984565fdf5233a888c591dc16a7841d479813b56a86e2145a239048"}
Jun 15 05:48:53.833377 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:53.833347    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wsmkh" event={"ID":"e586e4d3-72be-41a1-8b52-48301c1b36f4","Type":"ContainerStarted","Data":"d1f63866ac10fee4ff0c086c1c8a58c20e3170a977e0d47174b8d760d839fee2"}
Jun 15 05:48:56.443641 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.443612    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"]
Jun 15 05:48:56.445439 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.445416    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"
Jun 15 05:48:56.448997 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.448965    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-f6qjs\""
Jun 15 05:48:56.449699 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.449676    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jun 15 05:48:56.449699 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.449688    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jun 15 05:48:56.449877 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.449680    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jun 15 05:48:56.449877 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.449726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jun 15 05:48:56.451499 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.451478    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"]
Jun 15 05:48:56.453922 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.453572    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.456750 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.456728    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"]
Jun 15 05:48:56.457078 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.457062    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jun 15 05:48:56.457286 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.457240    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jun 15 05:48:56.457417 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.457402    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jun 15 05:48:56.458177 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.458155    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jun 15 05:48:56.473944 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.473916    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"]
Jun 15 05:48:56.489116 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.489074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb02772-1b27-4df2-af78-f832f27ca387-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.489309 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.489129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-ca\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.489309 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.489178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj656\" (UniqueName: \"kubernetes.io/projected/ddb02772-1b27-4df2-af78-f832f27ca387-kube-api-access-vj656\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.489309 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.489209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx6s9\" (UniqueName: \"kubernetes.io/projected/0dd06d13-a675-4360-9874-f9ae6ba602dd-kube-api-access-wx6s9\") pod \"managed-serviceaccount-addon-agent-64c685d9bd-gzq55\" (UID: \"0dd06d13-a675-4360-9874-f9ae6ba602dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"
Jun 15 05:48:56.489309 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.489234    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.489506 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.489307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-hub\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.489506 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.489381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0dd06d13-a675-4360-9874-f9ae6ba602dd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64c685d9bd-gzq55\" (UID: \"0dd06d13-a675-4360-9874-f9ae6ba602dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"
Jun 15 05:48:56.489506 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.489425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.590143 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.590103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb02772-1b27-4df2-af78-f832f27ca387-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.590374 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.590160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-ca\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.590374 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.590207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj656\" (UniqueName: \"kubernetes.io/projected/ddb02772-1b27-4df2-af78-f832f27ca387-kube-api-access-vj656\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.590374 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.590301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx6s9\" (UniqueName: \"kubernetes.io/projected/0dd06d13-a675-4360-9874-f9ae6ba602dd-kube-api-access-wx6s9\") pod \"managed-serviceaccount-addon-agent-64c685d9bd-gzq55\" (UID: \"0dd06d13-a675-4360-9874-f9ae6ba602dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"
Jun 15 05:48:56.590374 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.590348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.590552 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.590382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-hub\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.590552 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.590446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0dd06d13-a675-4360-9874-f9ae6ba602dd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64c685d9bd-gzq55\" (UID: \"0dd06d13-a675-4360-9874-f9ae6ba602dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"
Jun 15 05:48:56.590552 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.590471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.590956 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.590903    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb02772-1b27-4df2-af78-f832f27ca387-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.593388 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.593365    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-ca\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.594043 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.594002    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-hub\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.594043 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.594034    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.594192 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.594105    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0dd06d13-a675-4360-9874-f9ae6ba602dd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64c685d9bd-gzq55\" (UID: \"0dd06d13-a675-4360-9874-f9ae6ba602dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"
Jun 15 05:48:56.594192 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.594130    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddb02772-1b27-4df2-af78-f832f27ca387-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.599075 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.599036    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx6s9\" (UniqueName: \"kubernetes.io/projected/0dd06d13-a675-4360-9874-f9ae6ba602dd-kube-api-access-wx6s9\") pod \"managed-serviceaccount-addon-agent-64c685d9bd-gzq55\" (UID: \"0dd06d13-a675-4360-9874-f9ae6ba602dd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"
Jun 15 05:48:56.599161 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.599125    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj656\" (UniqueName: \"kubernetes.io/projected/ddb02772-1b27-4df2-af78-f832f27ca387-kube-api-access-vj656\") pod \"cluster-proxy-proxy-agent-7f6d7c89c9-h427d\" (UID: \"ddb02772-1b27-4df2-af78-f832f27ca387\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:56.761763 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.761728    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"
Jun 15 05:48:56.769807 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:56.769779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:48:58.229182 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:58.229149    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"]
Jun 15 05:48:58.232137 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:48:58.232087    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddb02772_1b27_4df2_af78_f832f27ca387.slice/crio-48ad79087ce2cd5e9c6ae3319d4e9d4ae11f78356bc598a91fa9dc1cb6f8fd94 WatchSource:0}: Error finding container 48ad79087ce2cd5e9c6ae3319d4e9d4ae11f78356bc598a91fa9dc1cb6f8fd94: Status 404 returned error can't find the container with id 48ad79087ce2cd5e9c6ae3319d4e9d4ae11f78356bc598a91fa9dc1cb6f8fd94
Jun 15 05:48:58.241428 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:58.241405    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55"]
Jun 15 05:48:58.244132 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:48:58.244108    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dd06d13_a675_4360_9874_f9ae6ba602dd.slice/crio-ad5fe7c42ac67334d319a133ec45a741210dc2bdadb9ba13539f8c3837816a78 WatchSource:0}: Error finding container ad5fe7c42ac67334d319a133ec45a741210dc2bdadb9ba13539f8c3837816a78: Status 404 returned error can't find the container with id ad5fe7c42ac67334d319a133ec45a741210dc2bdadb9ba13539f8c3837816a78
Jun 15 05:48:58.850481 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:58.850438    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2czf6" event={"ID":"6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53","Type":"ContainerStarted","Data":"f3797b54f9754ef87bc349688ae16def57e638e8d6146d8b9fb0922ddc29218c"}
Jun 15 05:48:58.851901 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:58.851868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" event={"ID":"ddb02772-1b27-4df2-af78-f832f27ca387","Type":"ContainerStarted","Data":"48ad79087ce2cd5e9c6ae3319d4e9d4ae11f78356bc598a91fa9dc1cb6f8fd94"}
Jun 15 05:48:58.853843 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:58.853773    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wsmkh" event={"ID":"e586e4d3-72be-41a1-8b52-48301c1b36f4","Type":"ContainerStarted","Data":"e320462bf19d4175f59c46b6adbef154d15921918907ac9bab39ecca385181cd"}
Jun 15 05:48:58.853994 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:58.853936    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:48:58.855556 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:58.855510    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55" event={"ID":"0dd06d13-a675-4360-9874-f9ae6ba602dd","Type":"ContainerStarted","Data":"ad5fe7c42ac67334d319a133ec45a741210dc2bdadb9ba13539f8c3837816a78"}
Jun 15 05:48:58.867419 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:58.867331    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2czf6" podStartSLOduration=67.50452698 podStartE2EDuration="1m11.867314023s" totalImagesPullingTime="4.362787043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:58.867291192 +0000 UTC m=+71.775171706" watchObservedRunningTime="2026-06-15 05:48:58.867314023 +0000 UTC m=+71.775194539"
Jun 15 05:48:58.883105 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:48:58.883036    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wsmkh" podStartSLOduration=67.557525086 podStartE2EDuration="1m11.883018724s" totalImagesPullingTime="4.325493638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:58.882011019 +0000 UTC m=+71.789891533" watchObservedRunningTime="2026-06-15 05:48:58.883018724 +0000 UTC m=+71.790899240"
Jun 15 05:49:02.869906 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:49:02.869867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55" event={"ID":"0dd06d13-a675-4360-9874-f9ae6ba602dd","Type":"ContainerStarted","Data":"2a792171e7080faea855af100ac06ae72388b9a970d403f41fe01eb042752ed8"}
Jun 15 05:49:02.871848 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:49:02.871826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" event={"ID":"ddb02772-1b27-4df2-af78-f832f27ca387","Type":"ContainerStarted","Data":"47b23164a3b6bfdf0a8934425a7b88eaaae2562651b2f35d678edebfadc946d9"}
Jun 15 05:49:02.871927 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:49:02.871859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" event={"ID":"ddb02772-1b27-4df2-af78-f832f27ca387","Type":"ContainerStarted","Data":"06daf3a3868235e2d2939c3d52e546a38e10c39d1b36aeca19c1c4cf0a224ebc"}
Jun 15 05:49:02.871927 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:49:02.871871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" event={"ID":"ddb02772-1b27-4df2-af78-f832f27ca387","Type":"ContainerStarted","Data":"169d6ee32315188ba597da1f34ba0bed66b17fee5cbe1df0c104ee5aa1d8f26c"}
Jun 15 05:49:02.889382 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:49:02.889311    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55" podStartSLOduration=3.26957578 podStartE2EDuration="6.889296721s" totalImagesPullingTime="3.619720941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:02.888523091 +0000 UTC m=+75.796403627" watchObservedRunningTime="2026-06-15 05:49:02.889296721 +0000 UTC m=+75.797177238"
Jun 15 05:49:02.909087 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:49:02.908909    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" podStartSLOduration=3.266026205 podStartE2EDuration="6.908892955s" totalImagesPullingTime="3.64286675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:02.90860642 +0000 UTC m=+75.816486938" watchObservedRunningTime="2026-06-15 05:49:02.908892955 +0000 UTC m=+75.816773467"
Jun 15 05:49:25.798499 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:49:25.798458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:49:25.798932 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:49:25.798623    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:49:25.799029 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:49:25.798897    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls podName:029b7613-6544-4c5b-aef8-3041d968a75f nodeName:}" failed. No retries permitted until 2026-06-15 05:50:29.798873624 +0000 UTC m=+162.706754144 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls") pod "dns-default-kn75r" (UID: "029b7613-6544-4c5b-aef8-3041d968a75f") : secret "dns-default-metrics-tls" not found
Jun 15 05:49:29.862573 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:49:29.862544    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wsmkh"
Jun 15 05:49:57.427932 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:49:57.427893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:49:57.428438 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:49:57.428026    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 15 05:49:57.428438 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:49:57.428087    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs podName:c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:59.428070618 +0000 UTC m=+252.335951132 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs") pod "network-metrics-daemon-cfxz2" (UID: "c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2") : secret "metrics-daemon-secret" not found
Jun 15 05:50:24.973763 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:50:24.973715    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kn75r" podUID="029b7613-6544-4c5b-aef8-3041d968a75f"
Jun 15 05:50:25.110751 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:50:25.110721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kn75r"
Jun 15 05:50:26.551134 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:50:26.551087    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cfxz2" podUID="c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2"
Jun 15 05:50:29.845620 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:50:29.845585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:50:29.846142 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:50:29.845741    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:50:29.846142 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:50:29.845828    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls podName:029b7613-6544-4c5b-aef8-3041d968a75f nodeName:}" failed. No retries permitted until 2026-06-15 05:52:31.845808266 +0000 UTC m=+284.753688770 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls") pod "dns-default-kn75r" (UID: "029b7613-6544-4c5b-aef8-3041d968a75f") : secret "dns-default-metrics-tls" not found
Jun 15 05:50:41.528048 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:50:41.527962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:51:02.217629 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:02.217588    2576 generic.go:356] "Generic (PLEG): container finished" podID="0dd06d13-a675-4360-9874-f9ae6ba602dd" containerID="2a792171e7080faea855af100ac06ae72388b9a970d403f41fe01eb042752ed8" exitCode=255
Jun 15 05:51:02.218165 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:02.217675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55" event={"ID":"0dd06d13-a675-4360-9874-f9ae6ba602dd","Type":"ContainerDied","Data":"2a792171e7080faea855af100ac06ae72388b9a970d403f41fe01eb042752ed8"}
Jun 15 05:51:02.218165 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:02.218078    2576 scope.go:117] "RemoveContainer" containerID="2a792171e7080faea855af100ac06ae72388b9a970d403f41fe01eb042752ed8"
Jun 15 05:51:03.221906 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:03.221870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c685d9bd-gzq55" event={"ID":"0dd06d13-a675-4360-9874-f9ae6ba602dd","Type":"ContainerStarted","Data":"d83716f2fa2e11078934a263b2bdffbe17f22d580866ce39a9f818b20b07261a"}
Jun 15 05:51:37.777971 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.777933    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sg2hl"]
Jun 15 05:51:37.779760 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.779745    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:37.782140 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.782114    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-snc9v\""
Jun 15 05:51:37.782304 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.782119    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 15 05:51:37.782924 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.782905    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 15 05:51:37.783036 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.782921    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 15 05:51:37.783036 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.782943    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 15 05:51:37.793294 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.793270    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sg2hl"]
Jun 15 05:51:37.921870 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.921831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-crio-socket\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:37.922044 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.921881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-data-volume\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:37.922044 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.921943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:37.922044 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.921990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66rq\" (UniqueName: \"kubernetes.io/projected/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-api-access-b66rq\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:37.922044 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:37.922040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.023088 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.023029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66rq\" (UniqueName: \"kubernetes.io/projected/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-api-access-b66rq\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.023299 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.023104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.023299 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.023138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-crio-socket\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.023299 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.023164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-data-volume\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.023299 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.023200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.023299 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:38.023279    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:38.023299 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.023289    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-crio-socket\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.023496 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:38.023309    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:38.023496 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:38.023357    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:38.523338074 +0000 UTC m=+231.431218578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:38.023496 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:38.023371    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:38.523365079 +0000 UTC m=+231.431245570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:38.023598 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.023516    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-data-volume\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.032800 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.032738    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66rq\" (UniqueName: \"kubernetes.io/projected/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-api-access-b66rq\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.527045 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.527004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.527209 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:38.527072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:38.527209 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:38.527140    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:38.527209 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:38.527159    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:38.527209 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:38.527210    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:39.527196459 +0000 UTC m=+232.435076952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:38.527381 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:38.527222    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:39.527216368 +0000 UTC m=+232.435096858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:39.534702 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:39.534666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:39.535143 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:39.534780    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:39.535143 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:39.534818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:39.535143 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:39.534842    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:41.534827856 +0000 UTC m=+234.442708352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:39.535143 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:39.534920    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:39.535143 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:39.534989    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:41.534971674 +0000 UTC m=+234.442852181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:40.566777 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:40.566746    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pspvw_52ce0a05-4c82-4b03-8386-6ebdb0a6bae5/dns-node-resolver/0.log"
Jun 15 05:51:41.550258 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:41.550189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:41.550438 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:41.550287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:41.550438 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:41.550349    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:41.550438 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:41.550365    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:41.550438 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:41.550412    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:45.55039666 +0000 UTC m=+238.458277151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:41.550438 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:41.550427    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:45.550419793 +0000 UTC m=+238.458300284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:41.567319 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:41.567293    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5k6jr_ecd20280-4ff2-42de-b4e2-85cdf88de891/node-ca/0.log"
Jun 15 05:51:42.566341 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:42.566312    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hslrg_1e3e29ab-7a28-4dd1-8d55-a3115197a16c/serve-healthcheck-canary/0.log"
Jun 15 05:51:45.578452 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:45.578413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:45.578972 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:45.578469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:45.578972 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:45.578533    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:45.578972 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:45.578578    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:45.578972 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:45.578585    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:53.578572059 +0000 UTC m=+246.486452554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:45.578972 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:51:45.578655    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls podName:2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f nodeName:}" failed. No retries permitted until 2026-06-15 05:51:53.578636574 +0000 UTC m=+246.486517066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sg2hl" (UID: "2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:53.634903 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:53.634859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:53.635315 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:53.634915    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:53.635420 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:53.635403    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:53.637178 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:53.637156    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sg2hl\" (UID: \"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f\") " pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:53.691344 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:53.691314    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-snc9v\""
Jun 15 05:51:53.698738 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:53.698710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sg2hl"
Jun 15 05:51:53.824358 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:53.824263    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sg2hl"]
Jun 15 05:51:53.826929 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:51:53.826899    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d58ceb3_e0b2_41e8_abdc_e0e6c6ce015f.slice/crio-1ca60b2b0439e5fbf88b359ac75aee54e77a44efce650c05156010d390d4fea2 WatchSource:0}: Error finding container 1ca60b2b0439e5fbf88b359ac75aee54e77a44efce650c05156010d390d4fea2: Status 404 returned error can't find the container with id 1ca60b2b0439e5fbf88b359ac75aee54e77a44efce650c05156010d390d4fea2
Jun 15 05:51:54.368748 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:54.368708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sg2hl" event={"ID":"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f","Type":"ContainerStarted","Data":"1ca60b2b0439e5fbf88b359ac75aee54e77a44efce650c05156010d390d4fea2"}
Jun 15 05:51:57.379732 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:57.379644    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sg2hl" event={"ID":"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f","Type":"ContainerStarted","Data":"9839c704a17ae46c6d0925638c5572c96f875eaa466e04c645f9f71d73ca4d34"}
Jun 15 05:51:58.384772 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:58.384735    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sg2hl" event={"ID":"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f","Type":"ContainerStarted","Data":"353c8b2eac671b5104d8da3df838ef02a62023d5a05e44e8ef84d80e5e43ff34"}
Jun 15 05:51:59.481457 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:59.481412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:51:59.484056 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:59.484027    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2-metrics-certs\") pod \"network-metrics-daemon-cfxz2\" (UID: \"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2\") " pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:51:59.533664 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:59.533631    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8qp7w\""
Jun 15 05:51:59.539193 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:59.539160    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfxz2"
Jun 15 05:51:59.668472 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:51:59.668438    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cfxz2"]
Jun 15 05:51:59.671841 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:51:59.671812    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b82b73_9ed8_4501_9cb0_b82bb6c21cf2.slice/crio-006d07d553d7296e791743584e1921f317a7512f2dfb2df7032ba96f1e23c337 WatchSource:0}: Error finding container 006d07d553d7296e791743584e1921f317a7512f2dfb2df7032ba96f1e23c337: Status 404 returned error can't find the container with id 006d07d553d7296e791743584e1921f317a7512f2dfb2df7032ba96f1e23c337
Jun 15 05:52:00.394028 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:00.393991    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sg2hl" event={"ID":"2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f","Type":"ContainerStarted","Data":"a296de5372ff67e0a7e0635a6c2fbea8d0008f63e01beff024e9ec6c42fe453c"}
Jun 15 05:52:00.395226 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:00.395196    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfxz2" event={"ID":"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2","Type":"ContainerStarted","Data":"006d07d553d7296e791743584e1921f317a7512f2dfb2df7032ba96f1e23c337"}
Jun 15 05:52:00.418104 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:00.418048    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sg2hl" podStartSLOduration=17.781309595 podStartE2EDuration="23.418029369s" totalImagesPullingTime="5.636719774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:51:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:00.417136109 +0000 UTC m=+253.325016661" watchObservedRunningTime="2026-06-15 05:52:00.418029369 +0000 UTC m=+253.325909887"
Jun 15 05:52:01.038110 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.038075    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-694c955b9-vqxmq"]
Jun 15 05:52:01.040923 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.040907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-694c955b9-vqxmq"
Jun 15 05:52:01.044235 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.044208    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gks2m\""
Jun 15 05:52:01.044235 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.044217    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jun 15 05:52:01.044823 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.044808    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jun 15 05:52:01.051667 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.051644    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-694c955b9-vqxmq"]
Jun 15 05:52:01.093905 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.093869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgbj\" (UniqueName: \"kubernetes.io/projected/627f8f86-bf57-4a37-92d5-8933d5eafa31-kube-api-access-9zgbj\") pod \"downloads-694c955b9-vqxmq\" (UID: \"627f8f86-bf57-4a37-92d5-8933d5eafa31\") " pod="openshift-console/downloads-694c955b9-vqxmq"
Jun 15 05:52:01.195067 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.195032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgbj\" (UniqueName: \"kubernetes.io/projected/627f8f86-bf57-4a37-92d5-8933d5eafa31-kube-api-access-9zgbj\") pod \"downloads-694c955b9-vqxmq\" (UID: \"627f8f86-bf57-4a37-92d5-8933d5eafa31\") " pod="openshift-console/downloads-694c955b9-vqxmq"
Jun 15 05:52:01.206263 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.206218    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgbj\" (UniqueName: \"kubernetes.io/projected/627f8f86-bf57-4a37-92d5-8933d5eafa31-kube-api-access-9zgbj\") pod \"downloads-694c955b9-vqxmq\" (UID: \"627f8f86-bf57-4a37-92d5-8933d5eafa31\") " pod="openshift-console/downloads-694c955b9-vqxmq"
Jun 15 05:52:01.349801 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.349712    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-694c955b9-vqxmq"
Jun 15 05:52:01.401755 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.401716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfxz2" event={"ID":"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2","Type":"ContainerStarted","Data":"7792339b237563af17266b3dce413773572d636c4330285a3b00ea6a92002798"}
Jun 15 05:52:01.401921 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.401779    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfxz2" event={"ID":"c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2","Type":"ContainerStarted","Data":"ef5c80c7f7a81a29c915c869b4b5146cbbb90d9e456c09489f67aa2a30c6216a"}
Jun 15 05:52:01.448621 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.448560    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cfxz2" podStartSLOduration=253.512886446 podStartE2EDuration="4m14.448535998s" totalImagesPullingTime="935.649552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:01.448532193 +0000 UTC m=+254.356412717" watchObservedRunningTime="2026-06-15 05:52:01.448535998 +0000 UTC m=+254.356416512"
Jun 15 05:52:01.515005 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:01.514970    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-694c955b9-vqxmq"]
Jun 15 05:52:01.517965 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:52:01.517940    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627f8f86_bf57_4a37_92d5_8933d5eafa31.slice/crio-e875642773ce73fe2c45ca5b09ae0066a03e9cc9065ecc36b0e2e21ba56c8d1e WatchSource:0}: Error finding container e875642773ce73fe2c45ca5b09ae0066a03e9cc9065ecc36b0e2e21ba56c8d1e: Status 404 returned error can't find the container with id e875642773ce73fe2c45ca5b09ae0066a03e9cc9065ecc36b0e2e21ba56c8d1e
Jun 15 05:52:02.406264 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:02.406220    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-694c955b9-vqxmq" event={"ID":"627f8f86-bf57-4a37-92d5-8933d5eafa31","Type":"ContainerStarted","Data":"e875642773ce73fe2c45ca5b09ae0066a03e9cc9065ecc36b0e2e21ba56c8d1e"}
Jun 15 05:52:14.844500 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.844285    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"]
Jun 15 05:52:14.848690 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.848665    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:14.852769 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.852381    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jun 15 05:52:14.852769 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.852533    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jun 15 05:52:14.852769 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.852382    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9tgjw\""
Jun 15 05:52:14.852769 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.852723    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 15 05:52:14.852769 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.852738    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 15 05:52:14.852769 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.852736    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 15 05:52:14.861885 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.861847    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"]
Jun 15 05:52:14.865281 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.865241    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-td7mk"]
Jun 15 05:52:14.865434 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.865415    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:14.868438 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.868053    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jun 15 05:52:14.868438 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.868057    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4njrw\""
Jun 15 05:52:14.868438 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.868276    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jun 15 05:52:14.868438 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.868344    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jun 15 05:52:14.868438 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.868423    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"]
Jun 15 05:52:14.868765 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.868539    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:14.871226 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.871101    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 15 05:52:14.871226 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.871110    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zztp9\""
Jun 15 05:52:14.871226 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.871176    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 15 05:52:14.871226 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.871102    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 15 05:52:14.879110 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.879086    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"]
Jun 15 05:52:14.903894 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.903841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5c32636-2166-400e-b6bb-ca3f6c6de016-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:14.904068 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.903903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5c32636-2166-400e-b6bb-ca3f6c6de016-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:14.904068 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.903962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c32636-2166-400e-b6bb-ca3f6c6de016-metrics-client-ca\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:14.904068 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:14.903980    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744r7\" (UniqueName: \"kubernetes.io/projected/d5c32636-2166-400e-b6bb-ca3f6c6de016-kube-api-access-744r7\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.004985 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.004952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9062eb8d-3809-445d-a781-df948d8a9e44-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.005173 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.004999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5c32636-2166-400e-b6bb-ca3f6c6de016-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.005173 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005124    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.005173 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9062eb8d-3809-445d-a781-df948d8a9e44-volume-directive-shadow\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.005379 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-root\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.005379 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5c32636-2166-400e-b6bb-ca3f6c6de016-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.005379 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-wtmp\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.005527 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-sys\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.005527 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96q7\" (UniqueName: \"kubernetes.io/projected/9062eb8d-3809-445d-a781-df948d8a9e44-kube-api-access-w96q7\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.005527 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-tls\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.005527 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-metrics-client-ca\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.005714 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9062eb8d-3809-445d-a781-df948d8a9e44-kube-state-metrics-tls\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.005714 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9062eb8d-3809-445d-a781-df948d8a9e44-metrics-client-ca\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.005714 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hprbd\" (UniqueName: \"kubernetes.io/projected/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-kube-api-access-hprbd\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.005714 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-textfile\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.005714 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9062eb8d-3809-445d-a781-df948d8a9e44-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.005929 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-accelerators-collector-config\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.005929 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744r7\" (UniqueName: \"kubernetes.io/projected/d5c32636-2166-400e-b6bb-ca3f6c6de016-kube-api-access-744r7\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.005929 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.005832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c32636-2166-400e-b6bb-ca3f6c6de016-metrics-client-ca\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.006631 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.006595    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c32636-2166-400e-b6bb-ca3f6c6de016-metrics-client-ca\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.007843 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.007814    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5c32636-2166-400e-b6bb-ca3f6c6de016-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.008257 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.008225    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5c32636-2166-400e-b6bb-ca3f6c6de016-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.018869 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.018845    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-744r7\" (UniqueName: \"kubernetes.io/projected/d5c32636-2166-400e-b6bb-ca3f6c6de016-kube-api-access-744r7\") pod \"openshift-state-metrics-6b8d965dd5-g2kmc\" (UID: \"d5c32636-2166-400e-b6bb-ca3f6c6de016\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.106410 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9062eb8d-3809-445d-a781-df948d8a9e44-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.106410 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.106410 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9062eb8d-3809-445d-a781-df948d8a9e44-volume-directive-shadow\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.106696 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-root\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.106696 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-wtmp\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.106696 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-sys\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.106696 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96q7\" (UniqueName: \"kubernetes.io/projected/9062eb8d-3809-445d-a781-df948d8a9e44-kube-api-access-w96q7\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.106696 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106543    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-root\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.106696 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-tls\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.106696 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-metrics-client-ca\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.106999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9062eb8d-3809-445d-a781-df948d8a9e44-volume-directive-shadow\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.106999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9062eb8d-3809-445d-a781-df948d8a9e44-kube-state-metrics-tls\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.106999 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106987    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-wtmp\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.107150 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.106992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9062eb8d-3809-445d-a781-df948d8a9e44-metrics-client-ca\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.107150 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.107062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hprbd\" (UniqueName: \"kubernetes.io/projected/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-kube-api-access-hprbd\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.107150 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.107101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-textfile\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.107150 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.107132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9062eb8d-3809-445d-a781-df948d8a9e44-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.107376 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.107168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-accelerators-collector-config\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.107376 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.107286    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-metrics-client-ca\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.107376 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.107312    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-sys\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.107580 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.107557    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-textfile\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.107796 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.107704    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-accelerators-collector-config\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.107796 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.107752    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9062eb8d-3809-445d-a781-df948d8a9e44-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.108241 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.108220    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9062eb8d-3809-445d-a781-df948d8a9e44-metrics-client-ca\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.109395 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.109353    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.109597 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.109573    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-node-exporter-tls\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.109912 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.109886    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9062eb8d-3809-445d-a781-df948d8a9e44-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.110081 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.110059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9062eb8d-3809-445d-a781-df948d8a9e44-kube-state-metrics-tls\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.120815 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.120791    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hprbd\" (UniqueName: \"kubernetes.io/projected/ee61ea6a-ac85-4bb0-93c7-9a0cad05f775-kube-api-access-hprbd\") pod \"node-exporter-td7mk\" (UID: \"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775\") " pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:15.121866 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.121844    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96q7\" (UniqueName: \"kubernetes.io/projected/9062eb8d-3809-445d-a781-df948d8a9e44-kube-api-access-w96q7\") pod \"kube-state-metrics-678bc549d7-wxn9c\" (UID: \"9062eb8d-3809-445d-a781-df948d8a9e44\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.160679 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.160649    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"
Jun 15 05:52:15.177937 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.177907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"
Jun 15 05:52:15.184746 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:15.184724    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-td7mk"
Jun 15 05:52:16.972765 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.972727    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78d655695f-cdz6t"]
Jun 15 05:52:16.976355 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.976331    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:16.981175 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.980943    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jun 15 05:52:16.981175 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.980995    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jun 15 05:52:16.981175 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.981053    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jun 15 05:52:16.981175 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.981071    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jun 15 05:52:16.981175 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.981077    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fzqm4\""
Jun 15 05:52:16.981175 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.981089    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-br4s2pumo69ju\""
Jun 15 05:52:16.981632 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.981566    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jun 15 05:52:16.995188 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:16.995160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78d655695f-cdz6t"]
Jun 15 05:52:17.021801 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.021768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-tls\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.021801 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.021810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b99a1226-c323-4db5-a0e7-8193b0b20e30-metrics-client-ca\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.022024 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.021871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.022024 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.021919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.022024 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.021973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.022162 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.022028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.022162 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.022051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrm4w\" (UniqueName: \"kubernetes.io/projected/b99a1226-c323-4db5-a0e7-8193b0b20e30-kube-api-access-lrm4w\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.022162 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.022111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-grpc-tls\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.122709 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.122665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.122921 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.122736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.122921 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.122771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.122921 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.122841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.122921 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.122867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrm4w\" (UniqueName: \"kubernetes.io/projected/b99a1226-c323-4db5-a0e7-8193b0b20e30-kube-api-access-lrm4w\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.122921 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.122906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-grpc-tls\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.123223 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.122948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-tls\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.123223 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.122973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b99a1226-c323-4db5-a0e7-8193b0b20e30-metrics-client-ca\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.124857 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.124794    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b99a1226-c323-4db5-a0e7-8193b0b20e30-metrics-client-ca\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.125693 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.125668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.125865 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.125842    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.126006 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.125985    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-grpc-tls\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.126429 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.126390    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.126813 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.126789    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-tls\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.127125 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.127101    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b99a1226-c323-4db5-a0e7-8193b0b20e30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.135296 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.135270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrm4w\" (UniqueName: \"kubernetes.io/projected/b99a1226-c323-4db5-a0e7-8193b0b20e30-kube-api-access-lrm4w\") pod \"thanos-querier-78d655695f-cdz6t\" (UID: \"b99a1226-c323-4db5-a0e7-8193b0b20e30\") " pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.287529 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.287434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:17.320603 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:52:17.320555    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee61ea6a_ac85_4bb0_93c7_9a0cad05f775.slice/crio-e3a889b0fcc0db4cd65698625dd515ee54aa83e2e36f5605b836cc2c5f447be9 WatchSource:0}: Error finding container e3a889b0fcc0db4cd65698625dd515ee54aa83e2e36f5605b836cc2c5f447be9: Status 404 returned error can't find the container with id e3a889b0fcc0db4cd65698625dd515ee54aa83e2e36f5605b836cc2c5f447be9
Jun 15 05:52:17.457942 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.457896    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-td7mk" event={"ID":"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775","Type":"ContainerStarted","Data":"e3a889b0fcc0db4cd65698625dd515ee54aa83e2e36f5605b836cc2c5f447be9"}
Jun 15 05:52:17.472520 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.472486    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78d655695f-cdz6t"]
Jun 15 05:52:17.475371 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:52:17.475290    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99a1226_c323_4db5_a0e7_8193b0b20e30.slice/crio-250513deb11b7ae3cdb4ca6111220fcd4f327bfce2f18452eeaee1b4f82b9211 WatchSource:0}: Error finding container 250513deb11b7ae3cdb4ca6111220fcd4f327bfce2f18452eeaee1b4f82b9211: Status 404 returned error can't find the container with id 250513deb11b7ae3cdb4ca6111220fcd4f327bfce2f18452eeaee1b4f82b9211
Jun 15 05:52:17.690381 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.690301    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc"]
Jun 15 05:52:17.694235 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:17.694023    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c"]
Jun 15 05:52:17.695968 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:52:17.695937    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c32636_2166_400e_b6bb_ca3f6c6de016.slice/crio-684ed3ef9aff3bb117a71076a8a4564f08732c686eb1473c6e7fa6157fd221ea WatchSource:0}: Error finding container 684ed3ef9aff3bb117a71076a8a4564f08732c686eb1473c6e7fa6157fd221ea: Status 404 returned error can't find the container with id 684ed3ef9aff3bb117a71076a8a4564f08732c686eb1473c6e7fa6157fd221ea
Jun 15 05:52:17.699146 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:52:17.699117    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9062eb8d_3809_445d_a781_df948d8a9e44.slice/crio-8da052519c5c7d08f204e8d6e467b0834cb8d91da6863062e5b62c46d8fe2504 WatchSource:0}: Error finding container 8da052519c5c7d08f204e8d6e467b0834cb8d91da6863062e5b62c46d8fe2504: Status 404 returned error can't find the container with id 8da052519c5c7d08f204e8d6e467b0834cb8d91da6863062e5b62c46d8fe2504
Jun 15 05:52:18.464665 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:18.464631    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c" event={"ID":"9062eb8d-3809-445d-a781-df948d8a9e44","Type":"ContainerStarted","Data":"8da052519c5c7d08f204e8d6e467b0834cb8d91da6863062e5b62c46d8fe2504"}
Jun 15 05:52:18.468547 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:18.468366    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t" event={"ID":"b99a1226-c323-4db5-a0e7-8193b0b20e30","Type":"ContainerStarted","Data":"250513deb11b7ae3cdb4ca6111220fcd4f327bfce2f18452eeaee1b4f82b9211"}
Jun 15 05:52:18.470835 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:18.470765    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-694c955b9-vqxmq" event={"ID":"627f8f86-bf57-4a37-92d5-8933d5eafa31","Type":"ContainerStarted","Data":"80a9b2b63c736f57b8ee50a18ef9bb30c0e13f2b4d65750c7a59c9df9dcc2b47"}
Jun 15 05:52:18.471361 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:18.471204    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-694c955b9-vqxmq"
Jun 15 05:52:18.474085 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:18.474053    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc" event={"ID":"d5c32636-2166-400e-b6bb-ca3f6c6de016","Type":"ContainerStarted","Data":"18293593f77ed25976654d110e741d52a21a99e4f5209bc0979824fc9924d6e5"}
Jun 15 05:52:18.474203 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:18.474105    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc" event={"ID":"d5c32636-2166-400e-b6bb-ca3f6c6de016","Type":"ContainerStarted","Data":"23d7d5d538180792d09e05619eb091abc781cce95d898ea0a01392b54de57d42"}
Jun 15 05:52:18.474203 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:18.474132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc" event={"ID":"d5c32636-2166-400e-b6bb-ca3f6c6de016","Type":"ContainerStarted","Data":"684ed3ef9aff3bb117a71076a8a4564f08732c686eb1473c6e7fa6157fd221ea"}
Jun 15 05:52:18.480195 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:18.480174    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-694c955b9-vqxmq"
Jun 15 05:52:18.518078 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:18.514923    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-694c955b9-vqxmq" podStartSLOduration=1.6147092189999999 podStartE2EDuration="17.514902552s" totalImagesPullingTime="15.900193333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:18.49134926 +0000 UTC m=+271.399229774" watchObservedRunningTime="2026-06-15 05:52:18.514902552 +0000 UTC m=+271.422783066"
Jun 15 05:52:19.487715 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.487672    2576 generic.go:356] "Generic (PLEG): container finished" podID="ee61ea6a-ac85-4bb0-93c7-9a0cad05f775" containerID="26e82c4762a2f340d0a79d63fa4b6227af4edeb687df952b50afcbfdc4a3055e" exitCode=0
Jun 15 05:52:19.489121 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.488958    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-td7mk" event={"ID":"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775","Type":"ContainerDied","Data":"26e82c4762a2f340d0a79d63fa4b6227af4edeb687df952b50afcbfdc4a3055e"}
Jun 15 05:52:19.619586 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.619541    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"]
Jun 15 05:52:19.640039 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.640004    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"]
Jun 15 05:52:19.640220 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.640127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:19.643285 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.642634    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jun 15 05:52:19.643285 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.642893    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mcbkj\""
Jun 15 05:52:19.749408 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.749310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fabf544d-8e04-4f91-aaae-da7bc43d9ec3-monitoring-plugin-cert\") pod \"monitoring-plugin-589576dfc7-jwrw9\" (UID: \"fabf544d-8e04-4f91-aaae-da7bc43d9ec3\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:19.749598 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.749425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkwg\" (UniqueName: \"kubernetes.io/projected/fabf544d-8e04-4f91-aaae-da7bc43d9ec3-kube-api-access-qqkwg\") pod \"monitoring-plugin-589576dfc7-jwrw9\" (UID: \"fabf544d-8e04-4f91-aaae-da7bc43d9ec3\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:19.850645 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.850602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkwg\" (UniqueName: \"kubernetes.io/projected/fabf544d-8e04-4f91-aaae-da7bc43d9ec3-kube-api-access-qqkwg\") pod \"monitoring-plugin-589576dfc7-jwrw9\" (UID: \"fabf544d-8e04-4f91-aaae-da7bc43d9ec3\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:19.850830 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.850707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fabf544d-8e04-4f91-aaae-da7bc43d9ec3-monitoring-plugin-cert\") pod \"monitoring-plugin-589576dfc7-jwrw9\" (UID: \"fabf544d-8e04-4f91-aaae-da7bc43d9ec3\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:19.850830 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:52:19.850825    2576 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jun 15 05:52:19.850942 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:52:19.850886    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabf544d-8e04-4f91-aaae-da7bc43d9ec3-monitoring-plugin-cert podName:fabf544d-8e04-4f91-aaae-da7bc43d9ec3 nodeName:}" failed. No retries permitted until 2026-06-15 05:52:20.350870394 +0000 UTC m=+273.258750886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/fabf544d-8e04-4f91-aaae-da7bc43d9ec3-monitoring-plugin-cert") pod "monitoring-plugin-589576dfc7-jwrw9" (UID: "fabf544d-8e04-4f91-aaae-da7bc43d9ec3") : secret "monitoring-plugin-cert" not found
Jun 15 05:52:19.860847 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:19.860807    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkwg\" (UniqueName: \"kubernetes.io/projected/fabf544d-8e04-4f91-aaae-da7bc43d9ec3-kube-api-access-qqkwg\") pod \"monitoring-plugin-589576dfc7-jwrw9\" (UID: \"fabf544d-8e04-4f91-aaae-da7bc43d9ec3\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:20.356488 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:20.356443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fabf544d-8e04-4f91-aaae-da7bc43d9ec3-monitoring-plugin-cert\") pod \"monitoring-plugin-589576dfc7-jwrw9\" (UID: \"fabf544d-8e04-4f91-aaae-da7bc43d9ec3\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:20.359467 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:20.359432    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fabf544d-8e04-4f91-aaae-da7bc43d9ec3-monitoring-plugin-cert\") pod \"monitoring-plugin-589576dfc7-jwrw9\" (UID: \"fabf544d-8e04-4f91-aaae-da7bc43d9ec3\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:20.553707 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:20.553449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:21.344022 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.343940    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"]
Jun 15 05:52:21.348537 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:52:21.348496    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabf544d_8e04_4f91_aaae_da7bc43d9ec3.slice/crio-a2e3d4462aaca3f88b86632b6c61df17de76aeb9869af9f4471a2b0313b62663 WatchSource:0}: Error finding container a2e3d4462aaca3f88b86632b6c61df17de76aeb9869af9f4471a2b0313b62663: Status 404 returned error can't find the container with id a2e3d4462aaca3f88b86632b6c61df17de76aeb9869af9f4471a2b0313b62663
Jun 15 05:52:21.505057 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.504926    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c" event={"ID":"9062eb8d-3809-445d-a781-df948d8a9e44","Type":"ContainerStarted","Data":"e51bf397b21d379bdccba8a05f03f2398592796f660a9530e51de6bb5f9e065d"}
Jun 15 05:52:21.505057 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.505003    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c" event={"ID":"9062eb8d-3809-445d-a781-df948d8a9e44","Type":"ContainerStarted","Data":"d2c0c0cfc436e38a5bf7ce76c2c8b642b296b51453f47a5660c89e384f9d672a"}
Jun 15 05:52:21.510403 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.510210    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t" event={"ID":"b99a1226-c323-4db5-a0e7-8193b0b20e30","Type":"ContainerStarted","Data":"9e50bdab674b7134e806fc1a2d6235762546badad316adc5457f9dc9c2a2f5d0"}
Jun 15 05:52:21.510403 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.510313    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t" event={"ID":"b99a1226-c323-4db5-a0e7-8193b0b20e30","Type":"ContainerStarted","Data":"481fb856ad5525bdeee2b3ff86ea7c602dd9c4f3d87fd12b622dc7ddd00e47f5"}
Jun 15 05:52:21.511779 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.511704    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9" event={"ID":"fabf544d-8e04-4f91-aaae-da7bc43d9ec3","Type":"ContainerStarted","Data":"a2e3d4462aaca3f88b86632b6c61df17de76aeb9869af9f4471a2b0313b62663"}
Jun 15 05:52:21.516110 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.514918    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-td7mk" event={"ID":"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775","Type":"ContainerStarted","Data":"fae8999045d200b0d2a6b400bf27b9f8771d3cbe5e3ef176086416b673961fc8"}
Jun 15 05:52:21.516110 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.514969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-td7mk" event={"ID":"ee61ea6a-ac85-4bb0-93c7-9a0cad05f775","Type":"ContainerStarted","Data":"4f696808fc1ac62e952341e8a3e15d24c1f8642e2b5ebc6b1a1bcd251f2afa84"}
Jun 15 05:52:21.523884 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.523152    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc" event={"ID":"d5c32636-2166-400e-b6bb-ca3f6c6de016","Type":"ContainerStarted","Data":"ec5c28b11e962f05f42f85c4e914f23e246826abee918fcfa483badeb560724f"}
Jun 15 05:52:21.542695 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.542416    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-td7mk" podStartSLOduration=6.304410085 podStartE2EDuration="7.542394288s" totalImagesPullingTime="1.084077125s" totalInitContainerRuntime="153.907078ms" isStatefulPod=true podCreationTimestamp="2026-06-15 05:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:21.539646068 +0000 UTC m=+274.447526584" watchObservedRunningTime="2026-06-15 05:52:21.542394288 +0000 UTC m=+274.450274803"
Jun 15 05:52:21.562836 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:21.562730    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-g2kmc" podStartSLOduration=4.297416882 podStartE2EDuration="7.562710062s" totalImagesPullingTime="3.26529318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:21.562512866 +0000 UTC m=+274.470393413" watchObservedRunningTime="2026-06-15 05:52:21.562710062 +0000 UTC m=+274.470590579"
Jun 15 05:52:22.536733 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:22.536673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c" event={"ID":"9062eb8d-3809-445d-a781-df948d8a9e44","Type":"ContainerStarted","Data":"d7392b80dc124d6c4088aceb0a98bdc1f00b7c0a72733bb7bdb003965cfc412e"}
Jun 15 05:52:22.540887 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:22.540852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t" event={"ID":"b99a1226-c323-4db5-a0e7-8193b0b20e30","Type":"ContainerStarted","Data":"0dc05d59ba9e9cf9c48d52f4cb5e7b2faf4240f65b95c4da1bf429823e3817ac"}
Jun 15 05:52:22.560724 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:22.560641    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-678bc549d7-wxn9c" podStartSLOduration=5.133877213 podStartE2EDuration="8.560623059s" totalImagesPullingTime="3.426745846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:22.558212996 +0000 UTC m=+275.466093513" watchObservedRunningTime="2026-06-15 05:52:22.560623059 +0000 UTC m=+275.468503572"
Jun 15 05:52:24.552831 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:24.552794    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t" event={"ID":"b99a1226-c323-4db5-a0e7-8193b0b20e30","Type":"ContainerStarted","Data":"8741dd7078126e167dababba1cda92e68804e04b81cdeca25f21ba351585678c"}
Jun 15 05:52:24.553446 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:24.552889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t" event={"ID":"b99a1226-c323-4db5-a0e7-8193b0b20e30","Type":"ContainerStarted","Data":"3348201df442c85ba7a9d3fbc157fc330a8bdbe8d16d17019079733746c12552"}
Jun 15 05:52:24.553446 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:24.552935    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t" event={"ID":"b99a1226-c323-4db5-a0e7-8193b0b20e30","Type":"ContainerStarted","Data":"d532748b5aca4bb6eae987fec107da888a38fe6cf9cdb5dc0c08dd6e61b4077e"}
Jun 15 05:52:24.553446 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:24.552984    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:24.554822 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:24.554791    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9" event={"ID":"fabf544d-8e04-4f91-aaae-da7bc43d9ec3","Type":"ContainerStarted","Data":"89bd6061af2f5c36c9cee76b31f8475be0aa913f76a53bdbea20e750838490ca"}
Jun 15 05:52:24.554967 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:24.554934    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:24.560433 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:24.560406    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9"
Jun 15 05:52:24.578615 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:24.578551    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t" podStartSLOduration=3.057002871 podStartE2EDuration="8.578528949s" totalImagesPullingTime="5.521526078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:52:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:24.57640949 +0000 UTC m=+277.484290003" watchObservedRunningTime="2026-06-15 05:52:24.578528949 +0000 UTC m=+277.486409465"
Jun 15 05:52:24.603508 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:24.603453    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-589576dfc7-jwrw9" podStartSLOduration=3.480308801 podStartE2EDuration="5.603436002s" totalImagesPullingTime="2.123127201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:24.602222889 +0000 UTC m=+277.510103429" watchObservedRunningTime="2026-06-15 05:52:24.603436002 +0000 UTC m=+277.511316530"
Jun 15 05:52:28.111496 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:52:28.111446    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kn75r" podUID="029b7613-6544-4c5b-aef8-3041d968a75f"
Jun 15 05:52:28.569741 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:28.569696    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kn75r"
Jun 15 05:52:30.565194 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:30.565157    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78d655695f-cdz6t"
Jun 15 05:52:31.864623 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:31.863682    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:52:31.868183 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:31.867685    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/029b7613-6544-4c5b-aef8-3041d968a75f-metrics-tls\") pod \"dns-default-kn75r\" (UID: \"029b7613-6544-4c5b-aef8-3041d968a75f\") " pod="openshift-dns/dns-default-kn75r"
Jun 15 05:52:31.872892 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:31.872865    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5mfr5\""
Jun 15 05:52:31.881190 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:31.881167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kn75r"
Jun 15 05:52:32.053017 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:32.052982    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kn75r"]
Jun 15 05:52:32.056014 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:52:32.055987    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029b7613_6544_4c5b_aef8_3041d968a75f.slice/crio-734003b6e7fd70e4b08fd319ac5ec7e8df811113b3120de631737bf0c8b122c6 WatchSource:0}: Error finding container 734003b6e7fd70e4b08fd319ac5ec7e8df811113b3120de631737bf0c8b122c6: Status 404 returned error can't find the container with id 734003b6e7fd70e4b08fd319ac5ec7e8df811113b3120de631737bf0c8b122c6
Jun 15 05:52:32.585219 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:32.585171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kn75r" event={"ID":"029b7613-6544-4c5b-aef8-3041d968a75f","Type":"ContainerStarted","Data":"734003b6e7fd70e4b08fd319ac5ec7e8df811113b3120de631737bf0c8b122c6"}
Jun 15 05:52:34.595599 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:34.595515    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kn75r" event={"ID":"029b7613-6544-4c5b-aef8-3041d968a75f","Type":"ContainerStarted","Data":"2772666c6ae5047384656b9761179012d4ff79518717fc52d7137ea6cb1cfe07"}
Jun 15 05:52:34.595599 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:34.595568    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kn75r" event={"ID":"029b7613-6544-4c5b-aef8-3041d968a75f","Type":"ContainerStarted","Data":"bf84efbb0e6f655af61dd065625212fdc91b5d646a8c015c677204b4ea989a7d"}
Jun 15 05:52:34.595981 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:34.595644    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kn75r"
Jun 15 05:52:34.622543 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:34.622495    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kn75r" podStartSLOduration=251.689579139 podStartE2EDuration="4m13.622478848s" totalImagesPullingTime="1.932899709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:34.620772692 +0000 UTC m=+287.528653215" watchObservedRunningTime="2026-06-15 05:52:34.622478848 +0000 UTC m=+287.530359358"
Jun 15 05:52:44.602326 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:44.602296    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kn75r"
Jun 15 05:52:47.486717 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:47.486692    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 05:52:47.492031 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:47.492005    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 05:52:47.495910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:52:47.495890    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jun 15 05:53:06.770943 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:53:06.770898    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" podUID="ddb02772-1b27-4df2-af78-f832f27ca387" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 15 05:53:16.771595 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:53:16.771547    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" podUID="ddb02772-1b27-4df2-af78-f832f27ca387" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 15 05:53:26.771016 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:53:26.770969    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" podUID="ddb02772-1b27-4df2-af78-f832f27ca387" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jun 15 05:53:26.771481 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:53:26.771050    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d"
Jun 15 05:53:26.771558 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:53:26.771525    2576 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"47b23164a3b6bfdf0a8934425a7b88eaaae2562651b2f35d678edebfadc946d9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jun 15 05:53:26.771601 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:53:26.771577    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" podUID="ddb02772-1b27-4df2-af78-f832f27ca387" containerName="service-proxy" containerID="cri-o://47b23164a3b6bfdf0a8934425a7b88eaaae2562651b2f35d678edebfadc946d9" gracePeriod=30
Jun 15 05:53:27.799838 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:53:27.799802    2576 generic.go:356] "Generic (PLEG): container finished" podID="ddb02772-1b27-4df2-af78-f832f27ca387" containerID="47b23164a3b6bfdf0a8934425a7b88eaaae2562651b2f35d678edebfadc946d9" exitCode=2
Jun 15 05:53:27.800217 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:53:27.799877    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" event={"ID":"ddb02772-1b27-4df2-af78-f832f27ca387","Type":"ContainerDied","Data":"47b23164a3b6bfdf0a8934425a7b88eaaae2562651b2f35d678edebfadc946d9"}
Jun 15 05:53:27.800217 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:53:27.799921    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6d7c89c9-h427d" event={"ID":"ddb02772-1b27-4df2-af78-f832f27ca387","Type":"ContainerStarted","Data":"bde11d85a8830e2dbd2f7f1cc66d6a8a80d356507c19236eda49d8070b7a801e"}
Jun 15 05:57:16.387166 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.387134    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-79wdh"]
Jun 15 05:57:16.389162 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.389143    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:57:16.394207 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.394181    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jun 15 05:57:16.395168 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.395141    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jun 15 05:57:16.395168 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.395162    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jun 15 05:57:16.395374 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.395164    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-7kpcf\""
Jun 15 05:57:16.421779 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.421739    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-79wdh"]
Jun 15 05:57:16.470187 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.470151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-992b4\" (UniqueName: \"kubernetes.io/projected/a2fef3c5-e630-4abf-8e4e-9ad07ef31813-kube-api-access-992b4\") pod \"llmisvc-controller-manager-84fc959947-79wdh\" (UID: \"a2fef3c5-e630-4abf-8e4e-9ad07ef31813\") " pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:57:16.470384 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.470198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fef3c5-e630-4abf-8e4e-9ad07ef31813-cert\") pod \"llmisvc-controller-manager-84fc959947-79wdh\" (UID: \"a2fef3c5-e630-4abf-8e4e-9ad07ef31813\") " pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:57:16.571604 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.571558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fef3c5-e630-4abf-8e4e-9ad07ef31813-cert\") pod \"llmisvc-controller-manager-84fc959947-79wdh\" (UID: \"a2fef3c5-e630-4abf-8e4e-9ad07ef31813\") " pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:57:16.571793 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.571636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992b4\" (UniqueName: \"kubernetes.io/projected/a2fef3c5-e630-4abf-8e4e-9ad07ef31813-kube-api-access-992b4\") pod \"llmisvc-controller-manager-84fc959947-79wdh\" (UID: \"a2fef3c5-e630-4abf-8e4e-9ad07ef31813\") " pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:57:16.573969 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.573948    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fef3c5-e630-4abf-8e4e-9ad07ef31813-cert\") pod \"llmisvc-controller-manager-84fc959947-79wdh\" (UID: \"a2fef3c5-e630-4abf-8e4e-9ad07ef31813\") " pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:57:16.581058 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.581029    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992b4\" (UniqueName: \"kubernetes.io/projected/a2fef3c5-e630-4abf-8e4e-9ad07ef31813-kube-api-access-992b4\") pod \"llmisvc-controller-manager-84fc959947-79wdh\" (UID: \"a2fef3c5-e630-4abf-8e4e-9ad07ef31813\") " pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:57:16.700259 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.700220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:57:16.849198 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.849162    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-79wdh"]
Jun 15 05:57:16.852647 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:57:16.852614    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda2fef3c5_e630_4abf_8e4e_9ad07ef31813.slice/crio-7b05d886a655034091d070d8f4e80cdc52ea342c9990ce55ec8fea5fbb03ac4b WatchSource:0}: Error finding container 7b05d886a655034091d070d8f4e80cdc52ea342c9990ce55ec8fea5fbb03ac4b: Status 404 returned error can't find the container with id 7b05d886a655034091d070d8f4e80cdc52ea342c9990ce55ec8fea5fbb03ac4b
Jun 15 05:57:16.853985 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:16.853966    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 05:57:17.622304 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:17.622270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-79wdh" event={"ID":"a2fef3c5-e630-4abf-8e4e-9ad07ef31813","Type":"ContainerStarted","Data":"7b05d886a655034091d070d8f4e80cdc52ea342c9990ce55ec8fea5fbb03ac4b"}
Jun 15 05:57:19.632474 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:19.632386    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-79wdh" event={"ID":"a2fef3c5-e630-4abf-8e4e-9ad07ef31813","Type":"ContainerStarted","Data":"89c81c5604c0e72a0662c4cbef350775cf49ca6942ae0e3a3efff0d0e84be6ce"}
Jun 15 05:57:19.632831 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:19.632484    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:57:19.668123 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:19.668065    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-79wdh" podStartSLOduration=1.245690746 podStartE2EDuration="3.668049465s" totalImagesPullingTime="2.422358719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:57:19.665932467 +0000 UTC m=+572.573812979" watchObservedRunningTime="2026-06-15 05:57:19.668049465 +0000 UTC m=+572.575929977"
Jun 15 05:57:47.516569 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:47.516531    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 05:57:47.517493 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:47.517472    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 05:57:50.638446 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:57:50.638404    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-79wdh"
Jun 15 05:58:25.643603 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.643569    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-prlg8"]
Jun 15 05:58:25.646578 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.646556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:25.648959 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.648938    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jun 15 05:58:25.649058 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.648965    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-b7plw\""
Jun 15 05:58:25.659818 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.659783    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-prlg8"]
Jun 15 05:58:25.713910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.713858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71b5b062-e119-49cc-a3a3-f74f6d3072cf-cert\") pod \"odh-model-controller-84b54b7f97-prlg8\" (UID: \"71b5b062-e119-49cc-a3a3-f74f6d3072cf\") " pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:25.713910 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.713909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvr4d\" (UniqueName: \"kubernetes.io/projected/71b5b062-e119-49cc-a3a3-f74f6d3072cf-kube-api-access-kvr4d\") pod \"odh-model-controller-84b54b7f97-prlg8\" (UID: \"71b5b062-e119-49cc-a3a3-f74f6d3072cf\") " pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:25.815106 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.815064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71b5b062-e119-49cc-a3a3-f74f6d3072cf-cert\") pod \"odh-model-controller-84b54b7f97-prlg8\" (UID: \"71b5b062-e119-49cc-a3a3-f74f6d3072cf\") " pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:25.815106 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.815107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvr4d\" (UniqueName: \"kubernetes.io/projected/71b5b062-e119-49cc-a3a3-f74f6d3072cf-kube-api-access-kvr4d\") pod \"odh-model-controller-84b54b7f97-prlg8\" (UID: \"71b5b062-e119-49cc-a3a3-f74f6d3072cf\") " pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:25.815358 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:58:25.815218    2576 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jun 15 05:58:25.815358 ip-10-0-129-86 kubenswrapper[2576]: E0615 05:58:25.815308    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71b5b062-e119-49cc-a3a3-f74f6d3072cf-cert podName:71b5b062-e119-49cc-a3a3-f74f6d3072cf nodeName:}" failed. No retries permitted until 2026-06-15 05:58:26.315292543 +0000 UTC m=+639.223173035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/71b5b062-e119-49cc-a3a3-f74f6d3072cf-cert") pod "odh-model-controller-84b54b7f97-prlg8" (UID: "71b5b062-e119-49cc-a3a3-f74f6d3072cf") : secret "odh-model-controller-webhook-cert" not found
Jun 15 05:58:25.825222 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:25.825195    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvr4d\" (UniqueName: \"kubernetes.io/projected/71b5b062-e119-49cc-a3a3-f74f6d3072cf-kube-api-access-kvr4d\") pod \"odh-model-controller-84b54b7f97-prlg8\" (UID: \"71b5b062-e119-49cc-a3a3-f74f6d3072cf\") " pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:26.319838 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:26.319797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71b5b062-e119-49cc-a3a3-f74f6d3072cf-cert\") pod \"odh-model-controller-84b54b7f97-prlg8\" (UID: \"71b5b062-e119-49cc-a3a3-f74f6d3072cf\") " pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:26.322227 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:26.322206    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71b5b062-e119-49cc-a3a3-f74f6d3072cf-cert\") pod \"odh-model-controller-84b54b7f97-prlg8\" (UID: \"71b5b062-e119-49cc-a3a3-f74f6d3072cf\") " pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:26.557389 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:26.557343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:26.687937 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:26.687756    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-prlg8"]
Jun 15 05:58:26.690540 ip-10-0-129-86 kubenswrapper[2576]: W0615 05:58:26.690512    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b5b062_e119_49cc_a3a3_f74f6d3072cf.slice/crio-f9252ae1f1ba1d5cdb4ade7c22d47ae0f9933460a431e674e418e3426f265ca7 WatchSource:0}: Error finding container f9252ae1f1ba1d5cdb4ade7c22d47ae0f9933460a431e674e418e3426f265ca7: Status 404 returned error can't find the container with id f9252ae1f1ba1d5cdb4ade7c22d47ae0f9933460a431e674e418e3426f265ca7
Jun 15 05:58:26.872892 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:26.872804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-prlg8" event={"ID":"71b5b062-e119-49cc-a3a3-f74f6d3072cf","Type":"ContainerStarted","Data":"f9252ae1f1ba1d5cdb4ade7c22d47ae0f9933460a431e674e418e3426f265ca7"}
Jun 15 05:58:29.890055 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:29.890018    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-prlg8" event={"ID":"71b5b062-e119-49cc-a3a3-f74f6d3072cf","Type":"ContainerStarted","Data":"f2a4904b518d8e12f88899ecc48df368148d9b947d0a16cdfa9bd9e9a35a4265"}
Jun 15 05:58:29.890476 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:29.890118    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 05:58:29.910906 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:29.910848    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-prlg8" podStartSLOduration=2.455412446 podStartE2EDuration="4.910827817s" totalImagesPullingTime="2.455415371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:58:29.910052028 +0000 UTC m=+642.817932540" watchObservedRunningTime="2026-06-15 05:58:29.910827817 +0000 UTC m=+642.818708326"
Jun 15 05:58:40.895950 ip-10-0-129-86 kubenswrapper[2576]: I0615 05:58:40.895916    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-prlg8"
Jun 15 06:00:12.178346 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.178309    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"]
Jun 15 06:00:12.181620 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.181599    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.183897 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.183874    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jun 15 06:00:12.185237 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.185220    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tjxr4\""
Jun 15 06:00:12.185377 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.185360    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 06:00:12.185911 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.185890    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jun 15 06:00:12.186036 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.185944    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 15 06:00:12.186036 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.186007    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jun 15 06:00:12.186036 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.186016    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 15 06:00:12.195807 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.195781    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"]
Jun 15 06:00:12.216545 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.216510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.216545 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.216552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.216743 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.216586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.216743 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.216653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2829m\" (UniqueName: \"kubernetes.io/projected/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-api-access-2829m\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.216743 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.216692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.317365 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.317328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.317365 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.317370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2829m\" (UniqueName: \"kubernetes.io/projected/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-api-access-2829m\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.317697 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.317390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.317697 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.317430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.317697 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.317458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.317888 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.317866    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.318159 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.318135    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.318271 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.318141    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.319841 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.319824    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.326676 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.326623    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2829m\" (UniqueName: \"kubernetes.io/projected/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-api-access-2829m\") pod \"isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.484406 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.484366    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"]
Jun 15 06:00:12.487943 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.487920    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.490285 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.490259    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5cf27-kube-rbac-proxy-sar-config\""
Jun 15 06:00:12.490394 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.490308    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5cf27-predictor-serving-cert\""
Jun 15 06:00:12.492044 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.492022    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:12.496133 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.496104    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"]
Jun 15 06:00:12.519069 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.519042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-proxy-tls\") pod \"error-404-isvc-5cf27-predictor-594bc754f5-k2pf4\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") " pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.519196 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.519097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5cf27-predictor-594bc754f5-k2pf4\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") " pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.519196 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.519132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-api-access-v42v4\") pod \"error-404-isvc-5cf27-predictor-594bc754f5-k2pf4\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") " pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.621272 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.620455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5cf27-predictor-594bc754f5-k2pf4\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") " pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.621272 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.620535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-api-access-v42v4\") pod \"error-404-isvc-5cf27-predictor-594bc754f5-k2pf4\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") " pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.621272 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.620608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-proxy-tls\") pod \"error-404-isvc-5cf27-predictor-594bc754f5-k2pf4\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") " pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.625266 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.621804    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5cf27-predictor-594bc754f5-k2pf4\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") " pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.626257 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.626202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-proxy-tls\") pod \"error-404-isvc-5cf27-predictor-594bc754f5-k2pf4\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") " pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.632508 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.632477    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"]
Jun 15 06:00:12.634199 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.634180    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-api-access-v42v4\") pod \"error-404-isvc-5cf27-predictor-594bc754f5-k2pf4\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") " pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.636097 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:00:12.636066    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3425ca6_4438_45c1_9f1f_dfb5d3a4acf6.slice/crio-973604109c14a2ba09d3222ee08447ce86b8dd8c2ed7aee042cfad574256d9b1 WatchSource:0}: Error finding container 973604109c14a2ba09d3222ee08447ce86b8dd8c2ed7aee042cfad574256d9b1: Status 404 returned error can't find the container with id 973604109c14a2ba09d3222ee08447ce86b8dd8c2ed7aee042cfad574256d9b1
Jun 15 06:00:12.799118 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.799020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:12.930486 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:12.930460    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"]
Jun 15 06:00:12.932901 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:00:12.932877    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce2a81e_0d68_49c4_a709_bdb5bffc8e42.slice/crio-614ed7a77e8f8609040076fbe3d5d32a111f952022d18686ab167dd72fd545eb WatchSource:0}: Error finding container 614ed7a77e8f8609040076fbe3d5d32a111f952022d18686ab167dd72fd545eb: Status 404 returned error can't find the container with id 614ed7a77e8f8609040076fbe3d5d32a111f952022d18686ab167dd72fd545eb
Jun 15 06:00:13.274323 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:13.274277    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" event={"ID":"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42","Type":"ContainerStarted","Data":"614ed7a77e8f8609040076fbe3d5d32a111f952022d18686ab167dd72fd545eb"}
Jun 15 06:00:13.275719 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:13.275690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" event={"ID":"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6","Type":"ContainerStarted","Data":"973604109c14a2ba09d3222ee08447ce86b8dd8c2ed7aee042cfad574256d9b1"}
Jun 15 06:00:26.345111 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:26.344993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" event={"ID":"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42","Type":"ContainerStarted","Data":"e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8"}
Jun 15 06:00:26.347265 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:26.347215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" event={"ID":"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6","Type":"ContainerStarted","Data":"f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24"}
Jun 15 06:00:28.357983 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:28.357948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" event={"ID":"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42","Type":"ContainerStarted","Data":"c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8"}
Jun 15 06:00:28.358384 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:28.358088    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:28.378408 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:28.378354    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" podStartSLOduration=1.1876010479999999 podStartE2EDuration="16.378337612s" totalImagesPullingTime="15.190736564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:00:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:00:28.377015187 +0000 UTC m=+761.284895703" watchObservedRunningTime="2026-06-15 06:00:28.378337612 +0000 UTC m=+761.286218125"
Jun 15 06:00:29.362720 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:29.362686    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:29.363935 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:29.363906    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 15 06:00:30.368603 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:30.368570    2576 generic.go:356] "Generic (PLEG): container finished" podID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerID="f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24" exitCode=0
Jun 15 06:00:30.369037 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:30.368646    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" event={"ID":"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6","Type":"ContainerDied","Data":"f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24"}
Jun 15 06:00:30.369037 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:30.368961    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 15 06:00:35.374293 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:35.374239    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:00:35.374873 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:35.374842    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 15 06:00:36.397138 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:36.397038    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" event={"ID":"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6","Type":"ContainerStarted","Data":"3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590"}
Jun 15 06:00:36.397626 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:36.397152    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" event={"ID":"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6","Type":"ContainerStarted","Data":"10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75"}
Jun 15 06:00:36.397626 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:36.397387    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:36.417288 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:36.417207    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podStartSLOduration=3.096790761 podStartE2EDuration="24.417183937s" totalImagesPullingTime="17.225084668s" totalInitContainerRuntime="4.095308508s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:00:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:00:36.415840244 +0000 UTC m=+769.323720760" watchObservedRunningTime="2026-06-15 06:00:36.417183937 +0000 UTC m=+769.325064452"
Jun 15 06:00:37.401579 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:37.401547    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:37.402719 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:37.402694    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:00:38.405743 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:38.405705    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:00:43.410728 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:43.410698    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:00:43.411411 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:43.411380    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:00:45.374930 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:45.374886    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 15 06:00:53.411742 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:53.411705    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:00:55.375340 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:00:55.375285    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 15 06:01:03.411718 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:03.411676    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:01:05.375828 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:05.375737    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jun 15 06:01:13.412303 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:13.412236    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:01:15.375963 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:15.375929    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:01:23.411663 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:23.411615    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:01:33.412303 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:33.412235    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:01:43.411806 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:43.411760    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:01:45.672968 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.672937    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"]
Jun 15 06:01:45.673420 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.673164    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container" containerID="cri-o://e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8" gracePeriod=30
Jun 15 06:01:45.673420 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.673217    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kube-rbac-proxy" containerID="cri-o://c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8" gracePeriod=30
Jun 15 06:01:45.691225 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.691198    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"]
Jun 15 06:01:45.693908 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.693890    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:45.696274 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.696215    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ea4bd-predictor-serving-cert\""
Jun 15 06:01:45.696514 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.696498    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ea4bd-kube-rbac-proxy-sar-config\""
Jun 15 06:01:45.705129 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.705107    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"]
Jun 15 06:01:45.790889 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.790864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ea4bd-predictor-776b445677-bptlg\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") " pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:45.790963 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.790908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbx79\" (UniqueName: \"kubernetes.io/projected/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-api-access-kbx79\") pod \"error-404-isvc-ea4bd-predictor-776b445677-bptlg\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") " pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:45.790963 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.790944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-proxy-tls\") pod \"error-404-isvc-ea4bd-predictor-776b445677-bptlg\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") " pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:45.891507 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.891462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ea4bd-predictor-776b445677-bptlg\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") " pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:45.891507 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.891514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbx79\" (UniqueName: \"kubernetes.io/projected/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-api-access-kbx79\") pod \"error-404-isvc-ea4bd-predictor-776b445677-bptlg\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") " pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:45.891712 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.891541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-proxy-tls\") pod \"error-404-isvc-ea4bd-predictor-776b445677-bptlg\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") " pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:45.892216 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.892192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ea4bd-predictor-776b445677-bptlg\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") " pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:45.894101 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.894078    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-proxy-tls\") pod \"error-404-isvc-ea4bd-predictor-776b445677-bptlg\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") " pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:45.900514 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:45.900493    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbx79\" (UniqueName: \"kubernetes.io/projected/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-api-access-kbx79\") pod \"error-404-isvc-ea4bd-predictor-776b445677-bptlg\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") " pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:46.004101 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.004054    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:46.143827 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.142921    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"]
Jun 15 06:01:46.146118 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:01:46.146086    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9417a4f0_50ea_4f07_ac8b_390b6b25ef9f.slice/crio-11801655afe07ae1a7f4a269313fe26284fd32f170a16cf0e74cedcb8081d005 WatchSource:0}: Error finding container 11801655afe07ae1a7f4a269313fe26284fd32f170a16cf0e74cedcb8081d005: Status 404 returned error can't find the container with id 11801655afe07ae1a7f4a269313fe26284fd32f170a16cf0e74cedcb8081d005
Jun 15 06:01:46.677571 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.677489    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerID="c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8" exitCode=2
Jun 15 06:01:46.677571 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.677559    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" event={"ID":"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42","Type":"ContainerDied","Data":"c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8"}
Jun 15 06:01:46.679300 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.679272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" event={"ID":"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f","Type":"ContainerStarted","Data":"e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c"}
Jun 15 06:01:46.679417 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.679334    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" event={"ID":"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f","Type":"ContainerStarted","Data":"c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a"}
Jun 15 06:01:46.679417 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.679366    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" event={"ID":"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f","Type":"ContainerStarted","Data":"11801655afe07ae1a7f4a269313fe26284fd32f170a16cf0e74cedcb8081d005"}
Jun 15 06:01:46.679417 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.679404    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:46.679525 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.679434    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:46.680897 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.680867    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 15 06:01:46.700049 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:46.700000    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podStartSLOduration=1.699984567 podStartE2EDuration="1.699984567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:01:46.698940319 +0000 UTC m=+839.606820831" watchObservedRunningTime="2026-06-15 06:01:46.699984567 +0000 UTC m=+839.607865079"
Jun 15 06:01:47.684209 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:47.684170    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 15 06:01:48.916148 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:48.916120    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:01:49.015234 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.015142    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-proxy-tls\") pod \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") "
Jun 15 06:01:49.015234 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.015192    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-api-access-v42v4\") pod \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") "
Jun 15 06:01:49.015482 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.015242    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-rbac-proxy-sar-config\") pod \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\" (UID: \"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42\") "
Jun 15 06:01:49.015702 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.015669    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" (UID: "1ce2a81e-0d68-49c4-a709-bdb5bffc8e42"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:01:49.017381 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.017355    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" (UID: "1ce2a81e-0d68-49c4-a709-bdb5bffc8e42"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:01:49.017469 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.017407    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-api-access-v42v4" (OuterVolumeSpecName: "kube-api-access-v42v4") pod "1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" (UID: "1ce2a81e-0d68-49c4-a709-bdb5bffc8e42"). InnerVolumeSpecName "kube-api-access-v42v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:01:49.116126 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.116084    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v42v4\" (UniqueName: \"kubernetes.io/projected/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-api-access-v42v4\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:01:49.116126 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.116119    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:01:49.116126 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.116130    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:01:49.694142 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.694108    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerID="e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8" exitCode=0
Jun 15 06:01:49.694333 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.694155    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"
Jun 15 06:01:49.694333 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.694197    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" event={"ID":"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42","Type":"ContainerDied","Data":"e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8"}
Jun 15 06:01:49.694333 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.694267    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4" event={"ID":"1ce2a81e-0d68-49c4-a709-bdb5bffc8e42","Type":"ContainerDied","Data":"614ed7a77e8f8609040076fbe3d5d32a111f952022d18686ab167dd72fd545eb"}
Jun 15 06:01:49.694333 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.694307    2576 scope.go:117] "RemoveContainer" containerID="c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8"
Jun 15 06:01:49.702833 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.702815    2576 scope.go:117] "RemoveContainer" containerID="e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8"
Jun 15 06:01:49.713238 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.713200    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"]
Jun 15 06:01:49.713323 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.713266    2576 scope.go:117] "RemoveContainer" containerID="c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8"
Jun 15 06:01:49.713559 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:01:49.713536    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8\": container with ID starting with c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8 not found: ID does not exist" containerID="c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8"
Jun 15 06:01:49.713634 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.713578    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8"} err="failed to get container status \"c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8\": rpc error: code = NotFound desc = could not find container \"c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8\": container with ID starting with c3e555f528034777ab32997d1e14bad6a60fa809442789f3c0eddaf63c4d44f8 not found: ID does not exist"
Jun 15 06:01:49.713634 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.713599    2576 scope.go:117] "RemoveContainer" containerID="e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8"
Jun 15 06:01:49.713914 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:01:49.713879    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8\": container with ID starting with e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8 not found: ID does not exist" containerID="e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8"
Jun 15 06:01:49.713992 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.713920    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8"} err="failed to get container status \"e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8\": rpc error: code = NotFound desc = could not find container \"e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8\": container with ID starting with e26654cfb3fe990920d0230cfbffdb6b15c085658e3974fcb340188d01aa42f8 not found: ID does not exist"
Jun 15 06:01:49.716016 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:49.715995    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5cf27-predictor-594bc754f5-k2pf4"]
Jun 15 06:01:51.533359 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:51.533316    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" path="/var/lib/kubelet/pods/1ce2a81e-0d68-49c4-a709-bdb5bffc8e42/volumes"
Jun 15 06:01:52.688532 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:52.688505    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:01:52.688967 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:52.688940    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 15 06:01:53.412463 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:01:53.412436    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:02:02.688952 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:02.688909    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 15 06:02:12.689612 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:12.689568    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 15 06:02:22.689576 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:22.689534    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 15 06:02:32.690345 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:32.690315    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:02:41.455425 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.455267    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"]
Jun 15 06:02:41.455908 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.455572    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" containerID="cri-o://10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75" gracePeriod=30
Jun 15 06:02:41.455980 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.455875    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kube-rbac-proxy" containerID="cri-o://3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590" gracePeriod=30
Jun 15 06:02:41.616731 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.616694    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"]
Jun 15 06:02:41.617101 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.617082    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container"
Jun 15 06:02:41.617194 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.617104    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container"
Jun 15 06:02:41.617194 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.617131    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kube-rbac-proxy"
Jun 15 06:02:41.617194 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.617141    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kube-rbac-proxy"
Jun 15 06:02:41.617525 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.617223    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kserve-container"
Jun 15 06:02:41.617525 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.617235    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce2a81e-0d68-49c4-a709-bdb5bffc8e42" containerName="kube-rbac-proxy"
Jun 15 06:02:41.619607 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.619582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:41.621916 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.621891    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0ac5a-predictor-serving-cert\""
Jun 15 06:02:41.622681 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.622657    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0ac5a-kube-rbac-proxy-sar-config\""
Jun 15 06:02:41.635905 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.635870    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"]
Jun 15 06:02:41.746618 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.746522    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71312792-836b-4f3d-85e2-d3ee2e19fd55-proxy-tls\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:41.746618 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.746573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:41.746618 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.746601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrn5\" (UniqueName: \"kubernetes.io/projected/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-api-access-tmrn5\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:41.847840 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.847797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71312792-836b-4f3d-85e2-d3ee2e19fd55-proxy-tls\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:41.848013 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.847851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:41.848013 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:02:41.847968    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-serving-cert: secret "error-404-isvc-0ac5a-predictor-serving-cert" not found
Jun 15 06:02:41.848097 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:02:41.848052    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71312792-836b-4f3d-85e2-d3ee2e19fd55-proxy-tls podName:71312792-836b-4f3d-85e2-d3ee2e19fd55 nodeName:}" failed. No retries permitted until 2026-06-15 06:02:42.34803021 +0000 UTC m=+895.255910715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71312792-836b-4f3d-85e2-d3ee2e19fd55-proxy-tls") pod "error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" (UID: "71312792-836b-4f3d-85e2-d3ee2e19fd55") : secret "error-404-isvc-0ac5a-predictor-serving-cert" not found
Jun 15 06:02:41.848097 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.847970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrn5\" (UniqueName: \"kubernetes.io/projected/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-api-access-tmrn5\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:41.848530 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.848511    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:41.858835 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.858807    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrn5\" (UniqueName: \"kubernetes.io/projected/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-api-access-tmrn5\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:41.903592 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.903556    2576 generic.go:356] "Generic (PLEG): container finished" podID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerID="3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590" exitCode=2
Jun 15 06:02:41.903757 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:41.903628    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" event={"ID":"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6","Type":"ContainerDied","Data":"3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590"}
Jun 15 06:02:42.351375 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.351340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71312792-836b-4f3d-85e2-d3ee2e19fd55-proxy-tls\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:42.353745 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.353716    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71312792-836b-4f3d-85e2-d3ee2e19fd55-proxy-tls\") pod \"error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") " pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:42.533701 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.533659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:42.668567 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.668541    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"]
Jun 15 06:02:42.671151 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:02:42.671126    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71312792_836b_4f3d_85e2_d3ee2e19fd55.slice/crio-cefbb785ba9d3f70309d4b0db5000d26b4331d6f35deb80c33b18b00944128dc WatchSource:0}: Error finding container cefbb785ba9d3f70309d4b0db5000d26b4331d6f35deb80c33b18b00944128dc: Status 404 returned error can't find the container with id cefbb785ba9d3f70309d4b0db5000d26b4331d6f35deb80c33b18b00944128dc
Jun 15 06:02:42.914641 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.914545    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" event={"ID":"71312792-836b-4f3d-85e2-d3ee2e19fd55","Type":"ContainerStarted","Data":"ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc"}
Jun 15 06:02:42.914779 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.914640    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" event={"ID":"71312792-836b-4f3d-85e2-d3ee2e19fd55","Type":"ContainerStarted","Data":"1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c"}
Jun 15 06:02:42.914779 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.914692    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:42.914779 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.914713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" event={"ID":"71312792-836b-4f3d-85e2-d3ee2e19fd55","Type":"ContainerStarted","Data":"cefbb785ba9d3f70309d4b0db5000d26b4331d6f35deb80c33b18b00944128dc"}
Jun 15 06:02:42.914779 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.914732    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:42.915868 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.915843    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 15 06:02:42.941580 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:42.941526    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podStartSLOduration=1.941509453 podStartE2EDuration="1.941509453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:02:42.939022188 +0000 UTC m=+895.846902725" watchObservedRunningTime="2026-06-15 06:02:42.941509453 +0000 UTC m=+895.849389965"
Jun 15 06:02:43.406434 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:43.406390    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jun 15 06:02:43.411606 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:43.411577    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jun 15 06:02:43.918368 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:43.918332    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 15 06:02:46.396945 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.396919    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:02:46.486729 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.486638    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2829m\" (UniqueName: \"kubernetes.io/projected/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-api-access-2829m\") pod \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") "
Jun 15 06:02:46.486729 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.486678    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-cabundle-cert\") pod \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") "
Jun 15 06:02:46.486729 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.486706    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-proxy-tls\") pod \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") "
Jun 15 06:02:46.486981 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.486757    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kserve-provision-location\") pod \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") "
Jun 15 06:02:46.486981 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.486799    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-rbac-proxy-sar-config\") pod \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\" (UID: \"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6\") "
Jun 15 06:02:46.487139 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.487101    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" (UID: "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:02:46.487302 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.487219    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" (UID: "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:02:46.487373 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.487298    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" (UID: "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:02:46.489375 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.489352    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" (UID: "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:02:46.489375 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.489364    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-api-access-2829m" (OuterVolumeSpecName: "kube-api-access-2829m") pod "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" (UID: "e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6"). InnerVolumeSpecName "kube-api-access-2829m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:02:46.587850 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.587813    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2829m\" (UniqueName: \"kubernetes.io/projected/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-api-access-2829m\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:02:46.587850 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.587844    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-cabundle-cert\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:02:46.587850 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.587857    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:02:46.588077 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.587867    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kserve-provision-location\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:02:46.588077 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.587880    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:02:46.932862 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.932770    2576 generic.go:356] "Generic (PLEG): container finished" podID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerID="10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75" exitCode=0
Jun 15 06:02:46.932862 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.932828    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"
Jun 15 06:02:46.932862 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.932847    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" event={"ID":"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6","Type":"ContainerDied","Data":"10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75"}
Jun 15 06:02:46.933094 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.932907    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h" event={"ID":"e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6","Type":"ContainerDied","Data":"973604109c14a2ba09d3222ee08447ce86b8dd8c2ed7aee042cfad574256d9b1"}
Jun 15 06:02:46.933094 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.932936    2576 scope.go:117] "RemoveContainer" containerID="3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590"
Jun 15 06:02:46.941717 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.941694    2576 scope.go:117] "RemoveContainer" containerID="10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75"
Jun 15 06:02:46.949822 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.949804    2576 scope.go:117] "RemoveContainer" containerID="f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24"
Jun 15 06:02:46.957814 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.957789    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"]
Jun 15 06:02:46.958202 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.958188    2576 scope.go:117] "RemoveContainer" containerID="3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590"
Jun 15 06:02:46.958516 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:02:46.958497    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590\": container with ID starting with 3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590 not found: ID does not exist" containerID="3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590"
Jun 15 06:02:46.958599 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.958525    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590"} err="failed to get container status \"3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590\": rpc error: code = NotFound desc = could not find container \"3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590\": container with ID starting with 3a04a0f8002f88b77f1ecad01f220ea422c0c2b1755fc157e9e5c12aecddd590 not found: ID does not exist"
Jun 15 06:02:46.958599 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.958545    2576 scope.go:117] "RemoveContainer" containerID="10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75"
Jun 15 06:02:46.958792 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:02:46.958778    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75\": container with ID starting with 10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75 not found: ID does not exist" containerID="10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75"
Jun 15 06:02:46.958830 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.958795    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75"} err="failed to get container status \"10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75\": rpc error: code = NotFound desc = could not find container \"10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75\": container with ID starting with 10ef04ed0b3e37acb93a3c812e3a7e204dce9819c0d37bd813e684146d253f75 not found: ID does not exist"
Jun 15 06:02:46.958830 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.958809    2576 scope.go:117] "RemoveContainer" containerID="f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24"
Jun 15 06:02:46.959038 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:02:46.959019    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24\": container with ID starting with f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24 not found: ID does not exist" containerID="f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24"
Jun 15 06:02:46.959085 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.959041    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24"} err="failed to get container status \"f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24\": rpc error: code = NotFound desc = could not find container \"f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24\": container with ID starting with f81dda9ce351a0edd8f5e0700435986176c9ef9e9fa960964dd636273191af24 not found: ID does not exist"
Jun 15 06:02:46.963266 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:46.963215    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-9d88d7b4b-ncg6h"]
Jun 15 06:02:47.537239 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:47.537202    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" path="/var/lib/kubelet/pods/e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6/volumes"
Jun 15 06:02:47.548232 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:47.548207    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:02:47.551976 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:47.551957    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:02:48.923356 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:48.923323    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:02:48.923893 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:48.923866    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 15 06:02:58.924364 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:02:58.924318    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 15 06:03:08.924487 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:03:08.924447    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 15 06:03:18.924662 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:03:18.924622    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 15 06:03:28.925167 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:03:28.925135    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:07:47.572375 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:07:47.572302    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:07:47.576884 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:07:47.576857    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:11:00.764546 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.764511    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"]
Jun 15 06:11:00.766933 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.764766    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container" containerID="cri-o://c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a" gracePeriod=30
Jun 15 06:11:00.766933 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.764836    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kube-rbac-proxy" containerID="cri-o://e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c" gracePeriod=30
Jun 15 06:11:00.786172 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.786145    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"]
Jun 15 06:11:00.786507 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.786492    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kube-rbac-proxy"
Jun 15 06:11:00.786553 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.786509    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kube-rbac-proxy"
Jun 15 06:11:00.786553 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.786525    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container"
Jun 15 06:11:00.786553 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.786531    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container"
Jun 15 06:11:00.786553 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.786540    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="storage-initializer"
Jun 15 06:11:00.786553 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.786546    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="storage-initializer"
Jun 15 06:11:00.786728 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.786600    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kube-rbac-proxy"
Jun 15 06:11:00.786728 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.786611    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3425ca6-4438-45c1-9f1f-dfb5d3a4acf6" containerName="kserve-container"
Jun 15 06:11:00.789699 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.789678    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:00.791756 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.791732    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78ee8-predictor-serving-cert\""
Jun 15 06:11:00.791876 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.791862    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78ee8-kube-rbac-proxy-sar-config\""
Jun 15 06:11:00.799810 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.799785    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"]
Jun 15 06:11:00.931236 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.931202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46869756-215b-41e3-8812-8ac102fbc07d-proxy-tls\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:00.931470 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.931319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5t4w\" (UniqueName: \"kubernetes.io/projected/46869756-215b-41e3-8812-8ac102fbc07d-kube-api-access-t5t4w\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:00.931470 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:00.931354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46869756-215b-41e3-8812-8ac102fbc07d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:01.032358 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.032230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5t4w\" (UniqueName: \"kubernetes.io/projected/46869756-215b-41e3-8812-8ac102fbc07d-kube-api-access-t5t4w\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:01.032358 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.032321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46869756-215b-41e3-8812-8ac102fbc07d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:01.032611 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.032408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46869756-215b-41e3-8812-8ac102fbc07d-proxy-tls\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:01.032611 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:11:01.032559    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-serving-cert: secret "error-404-isvc-78ee8-predictor-serving-cert" not found
Jun 15 06:11:01.032720 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:11:01.032627    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46869756-215b-41e3-8812-8ac102fbc07d-proxy-tls podName:46869756-215b-41e3-8812-8ac102fbc07d nodeName:}" failed. No retries permitted until 2026-06-15 06:11:01.532605583 +0000 UTC m=+1394.440486092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/46869756-215b-41e3-8812-8ac102fbc07d-proxy-tls") pod "error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" (UID: "46869756-215b-41e3-8812-8ac102fbc07d") : secret "error-404-isvc-78ee8-predictor-serving-cert" not found
Jun 15 06:11:01.033054 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.033028    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46869756-215b-41e3-8812-8ac102fbc07d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:01.041098 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.041070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5t4w\" (UniqueName: \"kubernetes.io/projected/46869756-215b-41e3-8812-8ac102fbc07d-kube-api-access-t5t4w\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:01.538017 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.537976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46869756-215b-41e3-8812-8ac102fbc07d-proxy-tls\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:01.540449 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.540424    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46869756-215b-41e3-8812-8ac102fbc07d-proxy-tls\") pod \"error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") " pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:01.701031 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.700983    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:01.816162 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.816073    2576 generic.go:356] "Generic (PLEG): container finished" podID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerID="e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c" exitCode=2
Jun 15 06:11:01.816526 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.816147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" event={"ID":"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f","Type":"ContainerDied","Data":"e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c"}
Jun 15 06:11:01.839138 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:01.839114    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"]
Jun 15 06:11:01.841424 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:11:01.841398    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46869756_215b_41e3_8812_8ac102fbc07d.slice/crio-f0bcc89ee1431c51c9b7ae6f6c32d5485b930438277b5093504d0cb1c189e358 WatchSource:0}: Error finding container f0bcc89ee1431c51c9b7ae6f6c32d5485b930438277b5093504d0cb1c189e358: Status 404 returned error can't find the container with id f0bcc89ee1431c51c9b7ae6f6c32d5485b930438277b5093504d0cb1c189e358
Jun 15 06:11:02.685440 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:02.685399    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jun 15 06:11:02.689759 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:02.689730    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jun 15 06:11:02.821842 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:02.821809    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" event={"ID":"46869756-215b-41e3-8812-8ac102fbc07d","Type":"ContainerStarted","Data":"e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6"}
Jun 15 06:11:02.822264 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:02.821866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" event={"ID":"46869756-215b-41e3-8812-8ac102fbc07d","Type":"ContainerStarted","Data":"6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853"}
Jun 15 06:11:02.822264 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:02.821888    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" event={"ID":"46869756-215b-41e3-8812-8ac102fbc07d","Type":"ContainerStarted","Data":"f0bcc89ee1431c51c9b7ae6f6c32d5485b930438277b5093504d0cb1c189e358"}
Jun 15 06:11:02.822264 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:02.821936    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:02.845091 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:02.845031    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" podStartSLOduration=2.845012655 podStartE2EDuration="2.845012655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:11:02.843618535 +0000 UTC m=+1395.751499039" watchObservedRunningTime="2026-06-15 06:11:02.845012655 +0000 UTC m=+1395.752893167"
Jun 15 06:11:03.826093 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:03.826058    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:03.827344 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:03.827315    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 15 06:11:04.109512 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.109484    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:11:04.154459 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.154427    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-proxy-tls\") pod \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") "
Jun 15 06:11:04.154459 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.154472    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-rbac-proxy-sar-config\") pod \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") "
Jun 15 06:11:04.154720 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.154492    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbx79\" (UniqueName: \"kubernetes.io/projected/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-api-access-kbx79\") pod \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\" (UID: \"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f\") "
Jun 15 06:11:04.154804 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.154781    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" (UID: "9417a4f0-50ea-4f07-ac8b-390b6b25ef9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:04.156537 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.156514    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-api-access-kbx79" (OuterVolumeSpecName: "kube-api-access-kbx79") pod "9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" (UID: "9417a4f0-50ea-4f07-ac8b-390b6b25ef9f"). InnerVolumeSpecName "kube-api-access-kbx79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:11:04.156588 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.156514    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" (UID: "9417a4f0-50ea-4f07-ac8b-390b6b25ef9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:11:04.255499 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.255456    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:11:04.255499 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.255490    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:11:04.255499 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.255501    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbx79\" (UniqueName: \"kubernetes.io/projected/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f-kube-api-access-kbx79\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:11:04.830900 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.830864    2576 generic.go:356] "Generic (PLEG): container finished" podID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerID="c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a" exitCode=0
Jun 15 06:11:04.831379 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.830914    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"
Jun 15 06:11:04.831379 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.830946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" event={"ID":"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f","Type":"ContainerDied","Data":"c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a"}
Jun 15 06:11:04.831379 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.831011    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg" event={"ID":"9417a4f0-50ea-4f07-ac8b-390b6b25ef9f","Type":"ContainerDied","Data":"11801655afe07ae1a7f4a269313fe26284fd32f170a16cf0e74cedcb8081d005"}
Jun 15 06:11:04.831379 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.831040    2576 scope.go:117] "RemoveContainer" containerID="e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c"
Jun 15 06:11:04.831640 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.831597    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 15 06:11:04.840441 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.840421    2576 scope.go:117] "RemoveContainer" containerID="c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a"
Jun 15 06:11:04.848194 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.848176    2576 scope.go:117] "RemoveContainer" containerID="e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c"
Jun 15 06:11:04.848500 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:11:04.848479    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c\": container with ID starting with e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c not found: ID does not exist" containerID="e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c"
Jun 15 06:11:04.848575 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.848508    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c"} err="failed to get container status \"e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c\": rpc error: code = NotFound desc = could not find container \"e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c\": container with ID starting with e2f2132d550522b4b872e3454cd5909c589d169d0c69ee94135791d81ded8d4c not found: ID does not exist"
Jun 15 06:11:04.848575 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.848526    2576 scope.go:117] "RemoveContainer" containerID="c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a"
Jun 15 06:11:04.848728 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:11:04.848713    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a\": container with ID starting with c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a not found: ID does not exist" containerID="c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a"
Jun 15 06:11:04.848765 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.848733    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a"} err="failed to get container status \"c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a\": rpc error: code = NotFound desc = could not find container \"c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a\": container with ID starting with c376b38c7c112c580dd2ce07b4a40b1e5bc79fafa8be3b464356ea99fab97a6a not found: ID does not exist"
Jun 15 06:11:04.865341 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.865309    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"]
Jun 15 06:11:04.872596 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:04.872573    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ea4bd-predictor-776b445677-bptlg"]
Jun 15 06:11:05.532982 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:05.532943    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" path="/var/lib/kubelet/pods/9417a4f0-50ea-4f07-ac8b-390b6b25ef9f/volumes"
Jun 15 06:11:09.836669 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:09.836640    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:09.837320 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:09.837289    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 15 06:11:19.837733 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:19.837691    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 15 06:11:29.837761 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:29.837716    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 15 06:11:39.837299 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:39.837194    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jun 15 06:11:49.838413 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:49.838375    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:11:56.462373 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.462329    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"]
Jun 15 06:11:56.462838 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.462693    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container"
Jun 15 06:11:56.462838 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.462710    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container"
Jun 15 06:11:56.462838 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.462729    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kube-rbac-proxy"
Jun 15 06:11:56.462838 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.462735    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kube-rbac-proxy"
Jun 15 06:11:56.462838 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.462787    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kserve-container"
Jun 15 06:11:56.462838 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.462800    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9417a4f0-50ea-4f07-ac8b-390b6b25ef9f" containerName="kube-rbac-proxy"
Jun 15 06:11:56.467195 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.467172    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:56.472436 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.472407    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-42a73-kube-rbac-proxy-sar-config\""
Jun 15 06:11:56.473141 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.473124    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-42a73-predictor-serving-cert\""
Jun 15 06:11:56.483952 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.483921    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"]
Jun 15 06:11:56.508224 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.508189    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"]
Jun 15 06:11:56.508766 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.508545    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kube-rbac-proxy" containerID="cri-o://ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc" gracePeriod=30
Jun 15 06:11:56.508766 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.508519    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container" containerID="cri-o://1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c" gracePeriod=30
Jun 15 06:11:56.541192 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.541148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:56.541192 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.541189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc99t\" (UniqueName: \"kubernetes.io/projected/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-api-access-nc99t\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:56.541409 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.541214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dce1cf9-097e-4686-8d0e-5fdef494f378-proxy-tls\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:56.641804 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.641762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dce1cf9-097e-4686-8d0e-5fdef494f378-proxy-tls\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:56.641978 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.641868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:56.641978 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.641893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc99t\" (UniqueName: \"kubernetes.io/projected/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-api-access-nc99t\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:56.641978 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:11:56.641916    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-42a73-predictor-serving-cert: secret "error-404-isvc-42a73-predictor-serving-cert" not found
Jun 15 06:11:56.642097 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:11:56.641988    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dce1cf9-097e-4686-8d0e-5fdef494f378-proxy-tls podName:6dce1cf9-097e-4686-8d0e-5fdef494f378 nodeName:}" failed. No retries permitted until 2026-06-15 06:11:57.141971593 +0000 UTC m=+1450.049852085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6dce1cf9-097e-4686-8d0e-5fdef494f378-proxy-tls") pod "error-404-isvc-42a73-predictor-7b975775c9-6w5jz" (UID: "6dce1cf9-097e-4686-8d0e-5fdef494f378") : secret "error-404-isvc-42a73-predictor-serving-cert" not found
Jun 15 06:11:56.642581 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.642562    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:56.650492 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:56.650461    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc99t\" (UniqueName: \"kubernetes.io/projected/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-api-access-nc99t\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:57.038463 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:57.038424    2576 generic.go:356] "Generic (PLEG): container finished" podID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerID="ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc" exitCode=2
Jun 15 06:11:57.038645 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:57.038489    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" event={"ID":"71312792-836b-4f3d-85e2-d3ee2e19fd55","Type":"ContainerDied","Data":"ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc"}
Jun 15 06:11:57.146497 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:57.146454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dce1cf9-097e-4686-8d0e-5fdef494f378-proxy-tls\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:57.148892 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:57.148866    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dce1cf9-097e-4686-8d0e-5fdef494f378-proxy-tls\") pod \"error-404-isvc-42a73-predictor-7b975775c9-6w5jz\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") " pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:57.377504 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:57.377407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:57.511555 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:57.511520    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"]
Jun 15 06:11:57.516224 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:11:57.516196    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dce1cf9_097e_4686_8d0e_5fdef494f378.slice/crio-4fc77121ec5ecae01f57c7ada4a099e4f51a3d9e7af935ae01795506716134e8 WatchSource:0}: Error finding container 4fc77121ec5ecae01f57c7ada4a099e4f51a3d9e7af935ae01795506716134e8: Status 404 returned error can't find the container with id 4fc77121ec5ecae01f57c7ada4a099e4f51a3d9e7af935ae01795506716134e8
Jun 15 06:11:58.044507 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:58.044472    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" event={"ID":"6dce1cf9-097e-4686-8d0e-5fdef494f378","Type":"ContainerStarted","Data":"0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc"}
Jun 15 06:11:58.044698 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:58.044535    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" event={"ID":"6dce1cf9-097e-4686-8d0e-5fdef494f378","Type":"ContainerStarted","Data":"aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0"}
Jun 15 06:11:58.044698 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:58.044556    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" event={"ID":"6dce1cf9-097e-4686-8d0e-5fdef494f378","Type":"ContainerStarted","Data":"4fc77121ec5ecae01f57c7ada4a099e4f51a3d9e7af935ae01795506716134e8"}
Jun 15 06:11:58.044698 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:58.044606    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:58.063861 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:58.063799    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" podStartSLOduration=2.063783449 podStartE2EDuration="2.063783449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:11:58.062084051 +0000 UTC m=+1450.969964564" watchObservedRunningTime="2026-06-15 06:11:58.063783449 +0000 UTC m=+1450.971664016"
Jun 15 06:11:58.919539 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:58.919495    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jun 15 06:11:58.923915 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:58.923883    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jun 15 06:11:59.048767 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:59.048728    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:11:59.050130 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:59.050106    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 15 06:11:59.959762 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:11:59.959736    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:12:00.053787 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.053693    2576 generic.go:356] "Generic (PLEG): container finished" podID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerID="1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c" exitCode=0
Jun 15 06:12:00.053787 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.053742    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"
Jun 15 06:12:00.053787 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.053776    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" event={"ID":"71312792-836b-4f3d-85e2-d3ee2e19fd55","Type":"ContainerDied","Data":"1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c"}
Jun 15 06:12:00.054017 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.053835    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2" event={"ID":"71312792-836b-4f3d-85e2-d3ee2e19fd55","Type":"ContainerDied","Data":"cefbb785ba9d3f70309d4b0db5000d26b4331d6f35deb80c33b18b00944128dc"}
Jun 15 06:12:00.054017 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.053863    2576 scope.go:117] "RemoveContainer" containerID="ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc"
Jun 15 06:12:00.054390 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.054360    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 15 06:12:00.061900 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.061882    2576 scope.go:117] "RemoveContainer" containerID="1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c"
Jun 15 06:12:00.069440 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.069420    2576 scope.go:117] "RemoveContainer" containerID="ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc"
Jun 15 06:12:00.069691 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:12:00.069670    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc\": container with ID starting with ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc not found: ID does not exist" containerID="ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc"
Jun 15 06:12:00.069743 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.069701    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc"} err="failed to get container status \"ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc\": rpc error: code = NotFound desc = could not find container \"ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc\": container with ID starting with ac55f670d9d78ae761359611b830e1bb0b0280a9d97db02bbf46cc0924d544fc not found: ID does not exist"
Jun 15 06:12:00.069743 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.069718    2576 scope.go:117] "RemoveContainer" containerID="1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c"
Jun 15 06:12:00.069941 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:12:00.069927    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c\": container with ID starting with 1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c not found: ID does not exist" containerID="1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c"
Jun 15 06:12:00.069999 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.069944    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c"} err="failed to get container status \"1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c\": rpc error: code = NotFound desc = could not find container \"1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c\": container with ID starting with 1eb0b873afacc02a47130c7b08fefdf1c4733f2c957a1042627176113237e06c not found: ID does not exist"
Jun 15 06:12:00.071204 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.071186    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmrn5\" (UniqueName: \"kubernetes.io/projected/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-api-access-tmrn5\") pod \"71312792-836b-4f3d-85e2-d3ee2e19fd55\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") "
Jun 15 06:12:00.071322 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.071237    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-rbac-proxy-sar-config\") pod \"71312792-836b-4f3d-85e2-d3ee2e19fd55\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") "
Jun 15 06:12:00.071322 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.071307    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71312792-836b-4f3d-85e2-d3ee2e19fd55-proxy-tls\") pod \"71312792-836b-4f3d-85e2-d3ee2e19fd55\" (UID: \"71312792-836b-4f3d-85e2-d3ee2e19fd55\") "
Jun 15 06:12:00.071768 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.071741    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71312792-836b-4f3d-85e2-d3ee2e19fd55" (UID: "71312792-836b-4f3d-85e2-d3ee2e19fd55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:12:00.073268 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.073226    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-api-access-tmrn5" (OuterVolumeSpecName: "kube-api-access-tmrn5") pod "71312792-836b-4f3d-85e2-d3ee2e19fd55" (UID: "71312792-836b-4f3d-85e2-d3ee2e19fd55"). InnerVolumeSpecName "kube-api-access-tmrn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:12:00.073358 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.073281    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71312792-836b-4f3d-85e2-d3ee2e19fd55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71312792-836b-4f3d-85e2-d3ee2e19fd55" (UID: "71312792-836b-4f3d-85e2-d3ee2e19fd55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:12:00.172935 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.172889    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrn5\" (UniqueName: \"kubernetes.io/projected/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-api-access-tmrn5\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:12:00.172935 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.172926    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71312792-836b-4f3d-85e2-d3ee2e19fd55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:12:00.172935 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.172936    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71312792-836b-4f3d-85e2-d3ee2e19fd55-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:12:00.377467 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.377434    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"]
Jun 15 06:12:00.381102 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:00.381074    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0ac5a-predictor-64c5f5c9bf-lfns2"]
Jun 15 06:12:01.533939 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:01.533906    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" path="/var/lib/kubelet/pods/71312792-836b-4f3d-85e2-d3ee2e19fd55/volumes"
Jun 15 06:12:05.060257 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:05.060223    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:12:05.060744 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:05.060679    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 15 06:12:11.031588 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.031550    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"]
Jun 15 06:12:11.032004 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.031861    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container" containerID="cri-o://6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853" gracePeriod=30
Jun 15 06:12:11.032004 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.031953    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kube-rbac-proxy" containerID="cri-o://e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6" gracePeriod=30
Jun 15 06:12:11.048567 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.048534    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"]
Jun 15 06:12:11.048860 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.048848    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kube-rbac-proxy"
Jun 15 06:12:11.048902 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.048861    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kube-rbac-proxy"
Jun 15 06:12:11.048902 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.048869    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container"
Jun 15 06:12:11.048902 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.048875    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container"
Jun 15 06:12:11.048990 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.048929    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kserve-container"
Jun 15 06:12:11.048990 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.048938    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="71312792-836b-4f3d-85e2-d3ee2e19fd55" containerName="kube-rbac-proxy"
Jun 15 06:12:11.053577 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.053551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.055765 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.055744    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b50b1-predictor-serving-cert\""
Jun 15 06:12:11.055866 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.055806    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b50b1-kube-rbac-proxy-sar-config\""
Jun 15 06:12:11.062378 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.062350    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"]
Jun 15 06:12:11.167195 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.167154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.167352 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.167205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-proxy-tls\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.167410 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.167340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjxc\" (UniqueName: \"kubernetes.io/projected/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-api-access-7pjxc\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.268541 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.268500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjxc\" (UniqueName: \"kubernetes.io/projected/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-api-access-7pjxc\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.268715 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.268558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.268715 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.268583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-proxy-tls\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.268715 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:12:11.268676    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-serving-cert: secret "error-404-isvc-b50b1-predictor-serving-cert" not found
Jun 15 06:12:11.268839 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:12:11.268760    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-proxy-tls podName:f9eb6f15-c14d-4b45-a14e-36d7b1275ff6 nodeName:}" failed. No retries permitted until 2026-06-15 06:12:11.768740242 +0000 UTC m=+1464.676620732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-proxy-tls") pod "error-404-isvc-b50b1-predictor-7bfb854997-r62dl" (UID: "f9eb6f15-c14d-4b45-a14e-36d7b1275ff6") : secret "error-404-isvc-b50b1-predictor-serving-cert" not found
Jun 15 06:12:11.269323 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.269302    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.277257 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.277213    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjxc\" (UniqueName: \"kubernetes.io/projected/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-api-access-7pjxc\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.772697 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.772654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-proxy-tls\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.775142 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.775120    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-proxy-tls\") pod \"error-404-isvc-b50b1-predictor-7bfb854997-r62dl\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") " pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:11.963854 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:11.963810    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:12.093826 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:12.093791    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"]
Jun 15 06:12:12.096605 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:12:12.096578    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9eb6f15_c14d_4b45_a14e_36d7b1275ff6.slice/crio-a2cd3e0baf73d2e4b2dfbdb7f3608b142dde5208bfabbe3e913b29fa46d28566 WatchSource:0}: Error finding container a2cd3e0baf73d2e4b2dfbdb7f3608b142dde5208bfabbe3e913b29fa46d28566: Status 404 returned error can't find the container with id a2cd3e0baf73d2e4b2dfbdb7f3608b142dde5208bfabbe3e913b29fa46d28566
Jun 15 06:12:12.105202 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:12.105175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" event={"ID":"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6","Type":"ContainerStarted","Data":"a2cd3e0baf73d2e4b2dfbdb7f3608b142dde5208bfabbe3e913b29fa46d28566"}
Jun 15 06:12:12.106845 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:12.106821    2576 generic.go:356] "Generic (PLEG): container finished" podID="46869756-215b-41e3-8812-8ac102fbc07d" containerID="e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6" exitCode=2
Jun 15 06:12:12.106945 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:12.106860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" event={"ID":"46869756-215b-41e3-8812-8ac102fbc07d","Type":"ContainerDied","Data":"e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6"}
Jun 15 06:12:13.112076 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:13.112037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" event={"ID":"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6","Type":"ContainerStarted","Data":"14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10"}
Jun 15 06:12:13.112493 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:13.112100    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" event={"ID":"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6","Type":"ContainerStarted","Data":"0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9"}
Jun 15 06:12:13.112493 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:13.112125    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:13.130707 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:13.130640    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" podStartSLOduration=2.13062386 podStartE2EDuration="2.13062386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:12:13.128892941 +0000 UTC m=+1466.036773454" watchObservedRunningTime="2026-06-15 06:12:13.13062386 +0000 UTC m=+1466.038504372"
Jun 15 06:12:14.116915 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.116881    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:14.118082 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.118057    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 15 06:12:14.382925 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.382898    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:12:14.497323 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.497287    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46869756-215b-41e3-8812-8ac102fbc07d-proxy-tls\") pod \"46869756-215b-41e3-8812-8ac102fbc07d\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") "
Jun 15 06:12:14.497323 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.497327    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5t4w\" (UniqueName: \"kubernetes.io/projected/46869756-215b-41e3-8812-8ac102fbc07d-kube-api-access-t5t4w\") pod \"46869756-215b-41e3-8812-8ac102fbc07d\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") "
Jun 15 06:12:14.497589 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.497355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46869756-215b-41e3-8812-8ac102fbc07d-kube-rbac-proxy-sar-config\") pod \"46869756-215b-41e3-8812-8ac102fbc07d\" (UID: \"46869756-215b-41e3-8812-8ac102fbc07d\") "
Jun 15 06:12:14.497767 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.497742    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46869756-215b-41e3-8812-8ac102fbc07d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "46869756-215b-41e3-8812-8ac102fbc07d" (UID: "46869756-215b-41e3-8812-8ac102fbc07d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:12:14.499427 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.499410    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46869756-215b-41e3-8812-8ac102fbc07d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46869756-215b-41e3-8812-8ac102fbc07d" (UID: "46869756-215b-41e3-8812-8ac102fbc07d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:12:14.499554 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.499534    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46869756-215b-41e3-8812-8ac102fbc07d-kube-api-access-t5t4w" (OuterVolumeSpecName: "kube-api-access-t5t4w") pod "46869756-215b-41e3-8812-8ac102fbc07d" (UID: "46869756-215b-41e3-8812-8ac102fbc07d"). InnerVolumeSpecName "kube-api-access-t5t4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:12:14.598677 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.598635    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46869756-215b-41e3-8812-8ac102fbc07d-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:12:14.598677 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.598670    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5t4w\" (UniqueName: \"kubernetes.io/projected/46869756-215b-41e3-8812-8ac102fbc07d-kube-api-access-t5t4w\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:12:14.598677 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:14.598680    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46869756-215b-41e3-8812-8ac102fbc07d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:12:15.061128 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.061084    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 15 06:12:15.122010 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.121976    2576 generic.go:356] "Generic (PLEG): container finished" podID="46869756-215b-41e3-8812-8ac102fbc07d" containerID="6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853" exitCode=0
Jun 15 06:12:15.122436 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.122027    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"
Jun 15 06:12:15.122436 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.122060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" event={"ID":"46869756-215b-41e3-8812-8ac102fbc07d","Type":"ContainerDied","Data":"6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853"}
Jun 15 06:12:15.122436 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.122123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v" event={"ID":"46869756-215b-41e3-8812-8ac102fbc07d","Type":"ContainerDied","Data":"f0bcc89ee1431c51c9b7ae6f6c32d5485b930438277b5093504d0cb1c189e358"}
Jun 15 06:12:15.122436 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.122170    2576 scope.go:117] "RemoveContainer" containerID="e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6"
Jun 15 06:12:15.122739 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.122712    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 15 06:12:15.131146 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.131123    2576 scope.go:117] "RemoveContainer" containerID="6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853"
Jun 15 06:12:15.146882 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.146852    2576 scope.go:117] "RemoveContainer" containerID="e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6"
Jun 15 06:12:15.147316 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:12:15.147285    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6\": container with ID starting with e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6 not found: ID does not exist" containerID="e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6"
Jun 15 06:12:15.147413 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.147325    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6"} err="failed to get container status \"e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6\": rpc error: code = NotFound desc = could not find container \"e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6\": container with ID starting with e72ce79cfd410a31ab859a7f43402a0455875a2dbef5d4311ba5e9c891b00cb6 not found: ID does not exist"
Jun 15 06:12:15.147413 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.147346    2576 scope.go:117] "RemoveContainer" containerID="6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853"
Jun 15 06:12:15.147675 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:12:15.147657    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853\": container with ID starting with 6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853 not found: ID does not exist" containerID="6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853"
Jun 15 06:12:15.147725 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.147682    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853"} err="failed to get container status \"6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853\": rpc error: code = NotFound desc = could not find container \"6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853\": container with ID starting with 6abe29f87e4168a69364cecbb76d7f723db5fe266bfc03c125177bafe9752853 not found: ID does not exist"
Jun 15 06:12:15.148260 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.148219    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"]
Jun 15 06:12:15.149623 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.149600    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ee8-predictor-69ddbf844c-r8w7v"]
Jun 15 06:12:15.534682 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:15.534645    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46869756-215b-41e3-8812-8ac102fbc07d" path="/var/lib/kubelet/pods/46869756-215b-41e3-8812-8ac102fbc07d/volumes"
Jun 15 06:12:20.127311 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:20.127272    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:12:20.127900 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:20.127800    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 15 06:12:25.061521 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:25.061466    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 15 06:12:30.128353 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:30.128304    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 15 06:12:35.061626 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:35.061582    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 15 06:12:40.128465 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:40.128424    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 15 06:12:45.062180 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:45.062147    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:12:47.597058 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:47.597027    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:12:47.602847 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:47.602823    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:12:50.128762 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:12:50.128720    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 15 06:13:00.128332 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:00.128301    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:13:06.575863 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.575758    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"]
Jun 15 06:13:06.576282 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.576030    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container" containerID="cri-o://aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0" gracePeriod=30
Jun 15 06:13:06.576282 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.576091    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kube-rbac-proxy" containerID="cri-o://0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc" gracePeriod=30
Jun 15 06:13:06.741584 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.741543    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"]
Jun 15 06:13:06.741960 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.741933    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kube-rbac-proxy"
Jun 15 06:13:06.742059 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.741962    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kube-rbac-proxy"
Jun 15 06:13:06.742059 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.741976    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container"
Jun 15 06:13:06.742059 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.741985    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container"
Jun 15 06:13:06.742217 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.742094    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kube-rbac-proxy"
Jun 15 06:13:06.742217 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.742109    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="46869756-215b-41e3-8812-8ac102fbc07d" containerName="kserve-container"
Jun 15 06:13:06.745137 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.745116    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:06.747320 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.747296    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-44470-kube-rbac-proxy-sar-config\""
Jun 15 06:13:06.747438 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.747332    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-44470-predictor-serving-cert\""
Jun 15 06:13:06.761070 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.761041    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"]
Jun 15 06:13:06.841531 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.841435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef50367-92c0-4c69-9612-2a4829727c07-proxy-tls\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:06.841531 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.841492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph66n\" (UniqueName: \"kubernetes.io/projected/5ef50367-92c0-4c69-9612-2a4829727c07-kube-api-access-ph66n\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:06.841718 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.841557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef50367-92c0-4c69-9612-2a4829727c07-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:06.942948 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.942901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph66n\" (UniqueName: \"kubernetes.io/projected/5ef50367-92c0-4c69-9612-2a4829727c07-kube-api-access-ph66n\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:06.942948 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.942957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef50367-92c0-4c69-9612-2a4829727c07-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:06.943216 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.943002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef50367-92c0-4c69-9612-2a4829727c07-proxy-tls\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:06.943216 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:13:06.943096    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-44470-predictor-serving-cert: secret "error-404-isvc-44470-predictor-serving-cert" not found
Jun 15 06:13:06.943216 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:13:06.943151    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef50367-92c0-4c69-9612-2a4829727c07-proxy-tls podName:5ef50367-92c0-4c69-9612-2a4829727c07 nodeName:}" failed. No retries permitted until 2026-06-15 06:13:07.443132592 +0000 UTC m=+1520.351013083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5ef50367-92c0-4c69-9612-2a4829727c07-proxy-tls") pod "error-404-isvc-44470-predictor-677b7d7455-tz4zt" (UID: "5ef50367-92c0-4c69-9612-2a4829727c07") : secret "error-404-isvc-44470-predictor-serving-cert" not found
Jun 15 06:13:06.943701 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.943679    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef50367-92c0-4c69-9612-2a4829727c07-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:06.957154 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:06.957121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph66n\" (UniqueName: \"kubernetes.io/projected/5ef50367-92c0-4c69-9612-2a4829727c07-kube-api-access-ph66n\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:07.331223 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:07.331176    2576 generic.go:356] "Generic (PLEG): container finished" podID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerID="0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc" exitCode=2
Jun 15 06:13:07.331409 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:07.331263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" event={"ID":"6dce1cf9-097e-4686-8d0e-5fdef494f378","Type":"ContainerDied","Data":"0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc"}
Jun 15 06:13:07.447598 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:07.447550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef50367-92c0-4c69-9612-2a4829727c07-proxy-tls\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:07.449934 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:07.449916    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef50367-92c0-4c69-9612-2a4829727c07-proxy-tls\") pod \"error-404-isvc-44470-predictor-677b7d7455-tz4zt\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") " pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:07.654903 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:07.654806    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:07.792072 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:07.792035    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"]
Jun 15 06:13:07.795224 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:13:07.795196    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef50367_92c0_4c69_9612_2a4829727c07.slice/crio-c053118dffe331bc1b34e73470c7e163656baa3a5f6b6d1a3283bb108920180b WatchSource:0}: Error finding container c053118dffe331bc1b34e73470c7e163656baa3a5f6b6d1a3283bb108920180b: Status 404 returned error can't find the container with id c053118dffe331bc1b34e73470c7e163656baa3a5f6b6d1a3283bb108920180b
Jun 15 06:13:08.338997 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:08.338959    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" event={"ID":"5ef50367-92c0-4c69-9612-2a4829727c07","Type":"ContainerStarted","Data":"8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7"}
Jun 15 06:13:08.339208 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:08.339016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" event={"ID":"5ef50367-92c0-4c69-9612-2a4829727c07","Type":"ContainerStarted","Data":"8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a"}
Jun 15 06:13:08.339208 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:08.339037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" event={"ID":"5ef50367-92c0-4c69-9612-2a4829727c07","Type":"ContainerStarted","Data":"c053118dffe331bc1b34e73470c7e163656baa3a5f6b6d1a3283bb108920180b"}
Jun 15 06:13:08.339208 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:08.339165    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:08.339370 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:08.339230    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:08.340262 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:08.340223    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 15 06:13:08.363945 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:08.363893    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podStartSLOduration=2.363875594 podStartE2EDuration="2.363875594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:13:08.36185451 +0000 UTC m=+1521.269735036" watchObservedRunningTime="2026-06-15 06:13:08.363875594 +0000 UTC m=+1521.271756109"
Jun 15 06:13:09.342794 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:09.342755    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 15 06:13:09.825750 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:09.825721    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:13:09.968860 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:09.968824    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc99t\" (UniqueName: \"kubernetes.io/projected/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-api-access-nc99t\") pod \"6dce1cf9-097e-4686-8d0e-5fdef494f378\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") "
Jun 15 06:13:09.969044 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:09.968909    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-rbac-proxy-sar-config\") pod \"6dce1cf9-097e-4686-8d0e-5fdef494f378\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") "
Jun 15 06:13:09.969044 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:09.968933    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dce1cf9-097e-4686-8d0e-5fdef494f378-proxy-tls\") pod \"6dce1cf9-097e-4686-8d0e-5fdef494f378\" (UID: \"6dce1cf9-097e-4686-8d0e-5fdef494f378\") "
Jun 15 06:13:09.969313 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:09.969282    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6dce1cf9-097e-4686-8d0e-5fdef494f378" (UID: "6dce1cf9-097e-4686-8d0e-5fdef494f378"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:13:09.971078 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:09.971053    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dce1cf9-097e-4686-8d0e-5fdef494f378-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6dce1cf9-097e-4686-8d0e-5fdef494f378" (UID: "6dce1cf9-097e-4686-8d0e-5fdef494f378"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:13:09.971159 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:09.971076    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-api-access-nc99t" (OuterVolumeSpecName: "kube-api-access-nc99t") pod "6dce1cf9-097e-4686-8d0e-5fdef494f378" (UID: "6dce1cf9-097e-4686-8d0e-5fdef494f378"). InnerVolumeSpecName "kube-api-access-nc99t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:13:10.069908 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.069868    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:13:10.069908 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.069902    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dce1cf9-097e-4686-8d0e-5fdef494f378-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:13:10.069908 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.069916    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc99t\" (UniqueName: \"kubernetes.io/projected/6dce1cf9-097e-4686-8d0e-5fdef494f378-kube-api-access-nc99t\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:13:10.347961 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.347867    2576 generic.go:356] "Generic (PLEG): container finished" podID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerID="aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0" exitCode=0
Jun 15 06:13:10.347961 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.347916    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"
Jun 15 06:13:10.347961 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.347951    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" event={"ID":"6dce1cf9-097e-4686-8d0e-5fdef494f378","Type":"ContainerDied","Data":"aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0"}
Jun 15 06:13:10.348457 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.348016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz" event={"ID":"6dce1cf9-097e-4686-8d0e-5fdef494f378","Type":"ContainerDied","Data":"4fc77121ec5ecae01f57c7ada4a099e4f51a3d9e7af935ae01795506716134e8"}
Jun 15 06:13:10.348457 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.348044    2576 scope.go:117] "RemoveContainer" containerID="0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc"
Jun 15 06:13:10.356980 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.356960    2576 scope.go:117] "RemoveContainer" containerID="aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0"
Jun 15 06:13:10.364357 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.364334    2576 scope.go:117] "RemoveContainer" containerID="0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc"
Jun 15 06:13:10.364613 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:13:10.364591    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc\": container with ID starting with 0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc not found: ID does not exist" containerID="0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc"
Jun 15 06:13:10.364675 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.364622    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc"} err="failed to get container status \"0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc\": rpc error: code = NotFound desc = could not find container \"0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc\": container with ID starting with 0e5bbe37af56718508009c59ca4ad4f9b91fcbd6400e0443d2da6fc8312b4dcc not found: ID does not exist"
Jun 15 06:13:10.364675 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.364641    2576 scope.go:117] "RemoveContainer" containerID="aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0"
Jun 15 06:13:10.364838 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:13:10.364824    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0\": container with ID starting with aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0 not found: ID does not exist" containerID="aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0"
Jun 15 06:13:10.364874 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.364842    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0"} err="failed to get container status \"aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0\": rpc error: code = NotFound desc = could not find container \"aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0\": container with ID starting with aa3aa127a2dc34f3438ed1d65a5b40170381c0e976f010f47f79b27085b19ec0 not found: ID does not exist"
Jun 15 06:13:10.368737 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.368709    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"]
Jun 15 06:13:10.372501 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:10.372475    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-42a73-predictor-7b975775c9-6w5jz"]
Jun 15 06:13:11.533400 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:11.533361    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" path="/var/lib/kubelet/pods/6dce1cf9-097e-4686-8d0e-5fdef494f378/volumes"
Jun 15 06:13:14.346983 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:14.346952    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:13:14.347549 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:14.347518    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 15 06:13:24.347562 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:24.347520    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 15 06:13:34.348439 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:34.348396    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 15 06:13:44.347508 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:44.347460    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 15 06:13:54.348998 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:13:54.348964    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:17:47.622483 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:17:47.622455    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:17:47.628584 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:17:47.628561    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:21:25.853396 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.853361    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"]
Jun 15 06:21:25.855894 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.853625    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container" containerID="cri-o://0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9" gracePeriod=30
Jun 15 06:21:25.855894 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.853634    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kube-rbac-proxy" containerID="cri-o://14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10" gracePeriod=30
Jun 15 06:21:25.940003 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.939967    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"]
Jun 15 06:21:25.940310 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.940296    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kube-rbac-proxy"
Jun 15 06:21:25.940310 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.940311    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kube-rbac-proxy"
Jun 15 06:21:25.940398 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.940324    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container"
Jun 15 06:21:25.940398 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.940329    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container"
Jun 15 06:21:25.940398 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.940388    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kube-rbac-proxy"
Jun 15 06:21:25.940398 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.940396    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dce1cf9-097e-4686-8d0e-5fdef494f378" containerName="kserve-container"
Jun 15 06:21:25.943314 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.943293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:25.945433 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.945413    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5c5b6-kube-rbac-proxy-sar-config\""
Jun 15 06:21:25.945539 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.945524    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5c5b6-predictor-serving-cert\""
Jun 15 06:21:25.953657 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:25.953633    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"]
Jun 15 06:21:26.004027 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.003986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/384b72c2-64bc-4ba4-baba-46973675935b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.004027 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.004030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkpx\" (UniqueName: \"kubernetes.io/projected/384b72c2-64bc-4ba4-baba-46973675935b-kube-api-access-tfkpx\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.004227 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.004057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/384b72c2-64bc-4ba4-baba-46973675935b-proxy-tls\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.105095 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.105008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/384b72c2-64bc-4ba4-baba-46973675935b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.105095 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.105048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkpx\" (UniqueName: \"kubernetes.io/projected/384b72c2-64bc-4ba4-baba-46973675935b-kube-api-access-tfkpx\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.105095 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.105069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/384b72c2-64bc-4ba4-baba-46973675935b-proxy-tls\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.105363 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:21:26.105187    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-serving-cert: secret "error-404-isvc-5c5b6-predictor-serving-cert" not found
Jun 15 06:21:26.105363 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:21:26.105264    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/384b72c2-64bc-4ba4-baba-46973675935b-proxy-tls podName:384b72c2-64bc-4ba4-baba-46973675935b nodeName:}" failed. No retries permitted until 2026-06-15 06:21:26.605228368 +0000 UTC m=+2019.513108863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/384b72c2-64bc-4ba4-baba-46973675935b-proxy-tls") pod "error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" (UID: "384b72c2-64bc-4ba4-baba-46973675935b") : secret "error-404-isvc-5c5b6-predictor-serving-cert" not found
Jun 15 06:21:26.105716 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.105695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/384b72c2-64bc-4ba4-baba-46973675935b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.115200 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.115177    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkpx\" (UniqueName: \"kubernetes.io/projected/384b72c2-64bc-4ba4-baba-46973675935b-kube-api-access-tfkpx\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.265746 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.265714    2576 generic.go:356] "Generic (PLEG): container finished" podID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerID="14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10" exitCode=2
Jun 15 06:21:26.265904 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.265788    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" event={"ID":"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6","Type":"ContainerDied","Data":"14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10"}
Jun 15 06:21:26.609832 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.609771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/384b72c2-64bc-4ba4-baba-46973675935b-proxy-tls\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.612162 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.612145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/384b72c2-64bc-4ba4-baba-46973675935b-proxy-tls\") pod \"error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") " pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.853627 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.853586    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:26.982302 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:26.982274    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"]
Jun 15 06:21:26.984675 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:21:26.984648    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384b72c2_64bc_4ba4_baba_46973675935b.slice/crio-be6f8c99da2ce6c57c525b8f4d612aef7d62956188c697b8d90c3804eee1e20b WatchSource:0}: Error finding container be6f8c99da2ce6c57c525b8f4d612aef7d62956188c697b8d90c3804eee1e20b: Status 404 returned error can't find the container with id be6f8c99da2ce6c57c525b8f4d612aef7d62956188c697b8d90c3804eee1e20b
Jun 15 06:21:27.271057 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:27.271020    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" event={"ID":"384b72c2-64bc-4ba4-baba-46973675935b","Type":"ContainerStarted","Data":"b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518"}
Jun 15 06:21:27.271225 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:27.271089    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" event={"ID":"384b72c2-64bc-4ba4-baba-46973675935b","Type":"ContainerStarted","Data":"b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909"}
Jun 15 06:21:27.271225 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:27.271122    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:27.271225 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:27.271149    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" event={"ID":"384b72c2-64bc-4ba4-baba-46973675935b","Type":"ContainerStarted","Data":"be6f8c99da2ce6c57c525b8f4d612aef7d62956188c697b8d90c3804eee1e20b"}
Jun 15 06:21:27.288943 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:27.288887    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podStartSLOduration=2.288867112 podStartE2EDuration="2.288867112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:21:27.288325367 +0000 UTC m=+2020.196205879" watchObservedRunningTime="2026-06-15 06:21:27.288867112 +0000 UTC m=+2020.196747626"
Jun 15 06:21:28.275540 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:28.275513    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:28.276736 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:28.276709    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 15 06:21:29.105581 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.105555    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:21:29.130934 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.130887    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-rbac-proxy-sar-config\") pod \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") "
Jun 15 06:21:29.131102 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.130963    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-proxy-tls\") pod \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") "
Jun 15 06:21:29.131102 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.131018    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pjxc\" (UniqueName: \"kubernetes.io/projected/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-api-access-7pjxc\") pod \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\" (UID: \"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6\") "
Jun 15 06:21:29.131331 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.131308    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" (UID: "f9eb6f15-c14d-4b45-a14e-36d7b1275ff6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:21:29.133092 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.133058    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" (UID: "f9eb6f15-c14d-4b45-a14e-36d7b1275ff6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:21:29.133216 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.133107    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-api-access-7pjxc" (OuterVolumeSpecName: "kube-api-access-7pjxc") pod "f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" (UID: "f9eb6f15-c14d-4b45-a14e-36d7b1275ff6"). InnerVolumeSpecName "kube-api-access-7pjxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:21:29.232516 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.232459    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:21:29.232516 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.232500    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pjxc\" (UniqueName: \"kubernetes.io/projected/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-api-access-7pjxc\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:21:29.232516 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.232510    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:21:29.280342 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.280300    2576 generic.go:356] "Generic (PLEG): container finished" podID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerID="0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9" exitCode=0
Jun 15 06:21:29.280779 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.280348    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"
Jun 15 06:21:29.280779 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.280382    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" event={"ID":"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6","Type":"ContainerDied","Data":"0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9"}
Jun 15 06:21:29.280779 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.280445    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl" event={"ID":"f9eb6f15-c14d-4b45-a14e-36d7b1275ff6","Type":"ContainerDied","Data":"a2cd3e0baf73d2e4b2dfbdb7f3608b142dde5208bfabbe3e913b29fa46d28566"}
Jun 15 06:21:29.280779 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.280474    2576 scope.go:117] "RemoveContainer" containerID="14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10"
Jun 15 06:21:29.281161 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.281132    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 15 06:21:29.289190 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.289172    2576 scope.go:117] "RemoveContainer" containerID="0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9"
Jun 15 06:21:29.297335 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.297317    2576 scope.go:117] "RemoveContainer" containerID="14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10"
Jun 15 06:21:29.297646 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:21:29.297625    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10\": container with ID starting with 14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10 not found: ID does not exist" containerID="14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10"
Jun 15 06:21:29.297720 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.297655    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10"} err="failed to get container status \"14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10\": rpc error: code = NotFound desc = could not find container \"14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10\": container with ID starting with 14fd1a03b699ed8c1cecdcc3b2a35a8d2c6f9f5c63ae83802668200e1c48cd10 not found: ID does not exist"
Jun 15 06:21:29.297720 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.297673    2576 scope.go:117] "RemoveContainer" containerID="0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9"
Jun 15 06:21:29.297898 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:21:29.297880    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9\": container with ID starting with 0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9 not found: ID does not exist" containerID="0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9"
Jun 15 06:21:29.297945 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.297904    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9"} err="failed to get container status \"0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9\": rpc error: code = NotFound desc = could not find container \"0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9\": container with ID starting with 0a3ce9c9a2ba30b9f961846bc958461683a37cdbc21402101c5729e023037bb9 not found: ID does not exist"
Jun 15 06:21:29.300387 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.300364    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"]
Jun 15 06:21:29.304353 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.304330    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b50b1-predictor-7bfb854997-r62dl"]
Jun 15 06:21:29.538059 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:29.537969    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" path="/var/lib/kubelet/pods/f9eb6f15-c14d-4b45-a14e-36d7b1275ff6/volumes"
Jun 15 06:21:34.285446 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:34.285414    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:21:34.285978 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:34.285942    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 15 06:21:44.286419 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:44.286373    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 15 06:21:54.286782 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:21:54.286739    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 15 06:22:04.285881 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:04.285791    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 15 06:22:14.286844 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:14.286813    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:22:21.727375 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.727330    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"]
Jun 15 06:22:21.728174 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.727626    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container" containerID="cri-o://8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a" gracePeriod=30
Jun 15 06:22:21.728174 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.727664    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kube-rbac-proxy" containerID="cri-o://8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7" gracePeriod=30
Jun 15 06:22:21.743041 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.743015    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"]
Jun 15 06:22:21.743386 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.743372    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container"
Jun 15 06:22:21.743435 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.743389    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container"
Jun 15 06:22:21.743435 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.743414    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kube-rbac-proxy"
Jun 15 06:22:21.743435 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.743419    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kube-rbac-proxy"
Jun 15 06:22:21.743527 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.743470    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kserve-container"
Jun 15 06:22:21.743527 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.743480    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9eb6f15-c14d-4b45-a14e-36d7b1275ff6" containerName="kube-rbac-proxy"
Jun 15 06:22:21.746527 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.746511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:21.748937 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.748913    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6c800-predictor-serving-cert\""
Jun 15 06:22:21.749044 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.748948    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6c800-kube-rbac-proxy-sar-config\""
Jun 15 06:22:21.757551 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.757530    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"]
Jun 15 06:22:21.759148 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.759118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5efa7977-0132-444b-8cd0-956adf38943c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:21.759270 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.759178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkggq\" (UniqueName: \"kubernetes.io/projected/5efa7977-0132-444b-8cd0-956adf38943c-kube-api-access-fkggq\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:21.759270 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.759225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5efa7977-0132-444b-8cd0-956adf38943c-proxy-tls\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:21.860533 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.860497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5efa7977-0132-444b-8cd0-956adf38943c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:21.860713 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.860557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkggq\" (UniqueName: \"kubernetes.io/projected/5efa7977-0132-444b-8cd0-956adf38943c-kube-api-access-fkggq\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:21.860713 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.860597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5efa7977-0132-444b-8cd0-956adf38943c-proxy-tls\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:21.860831 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:22:21.860715    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-6c800-predictor-serving-cert: secret "error-404-isvc-6c800-predictor-serving-cert" not found
Jun 15 06:22:21.860831 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:22:21.860777    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5efa7977-0132-444b-8cd0-956adf38943c-proxy-tls podName:5efa7977-0132-444b-8cd0-956adf38943c nodeName:}" failed. No retries permitted until 2026-06-15 06:22:22.360757031 +0000 UTC m=+2075.268637539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5efa7977-0132-444b-8cd0-956adf38943c-proxy-tls") pod "error-404-isvc-6c800-predictor-54776894c5-6zr7k" (UID: "5efa7977-0132-444b-8cd0-956adf38943c") : secret "error-404-isvc-6c800-predictor-serving-cert" not found
Jun 15 06:22:21.861158 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.861135    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5efa7977-0132-444b-8cd0-956adf38943c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:21.872276 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:21.872220    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkggq\" (UniqueName: \"kubernetes.io/projected/5efa7977-0132-444b-8cd0-956adf38943c-kube-api-access-fkggq\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:22.364286 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:22.364223    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5efa7977-0132-444b-8cd0-956adf38943c-proxy-tls\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:22.366614 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:22.366593    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5efa7977-0132-444b-8cd0-956adf38943c-proxy-tls\") pod \"error-404-isvc-6c800-predictor-54776894c5-6zr7k\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:22.489738 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:22.489705    2576 generic.go:356] "Generic (PLEG): container finished" podID="5ef50367-92c0-4c69-9612-2a4829727c07" containerID="8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7" exitCode=2
Jun 15 06:22:22.489901 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:22.489785    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" event={"ID":"5ef50367-92c0-4c69-9612-2a4829727c07","Type":"ContainerDied","Data":"8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7"}
Jun 15 06:22:22.657979 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:22.657881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:22.787881 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:22.787737    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"]
Jun 15 06:22:22.790344 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:22:22.790304    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efa7977_0132_444b_8cd0_956adf38943c.slice/crio-326f8f4afdcd33b4e4cc07cc02c1067259ee2c2c13a39ad2bef8d0f3216c5291 WatchSource:0}: Error finding container 326f8f4afdcd33b4e4cc07cc02c1067259ee2c2c13a39ad2bef8d0f3216c5291: Status 404 returned error can't find the container with id 326f8f4afdcd33b4e4cc07cc02c1067259ee2c2c13a39ad2bef8d0f3216c5291
Jun 15 06:22:23.495912 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:23.495869    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" event={"ID":"5efa7977-0132-444b-8cd0-956adf38943c","Type":"ContainerStarted","Data":"9aa4d9727041fec18a51979607a7b0421287ee4779653bec747439ffaaa7dc53"}
Jun 15 06:22:23.496114 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:23.495929    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" event={"ID":"5efa7977-0132-444b-8cd0-956adf38943c","Type":"ContainerStarted","Data":"c4a502135064ea2d7279f03217ff6f4ee7e59f94fe973234b5a788a79ee3a37e"}
Jun 15 06:22:23.496114 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:23.495950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" event={"ID":"5efa7977-0132-444b-8cd0-956adf38943c","Type":"ContainerStarted","Data":"326f8f4afdcd33b4e4cc07cc02c1067259ee2c2c13a39ad2bef8d0f3216c5291"}
Jun 15 06:22:23.496114 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:23.495974    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:23.515694 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:23.515639    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" podStartSLOduration=2.5156206169999997 podStartE2EDuration="2.515620617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:22:23.514065178 +0000 UTC m=+2076.421945692" watchObservedRunningTime="2026-06-15 06:22:23.515620617 +0000 UTC m=+2076.423501128"
Jun 15 06:22:24.343147 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:24.343097    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jun 15 06:22:24.347831 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:24.347802    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 15 06:22:24.500026 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:24.499992    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:24.501464 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:24.501433    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 15 06:22:25.086678 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.086651    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:22:25.183585 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.183492    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph66n\" (UniqueName: \"kubernetes.io/projected/5ef50367-92c0-4c69-9612-2a4829727c07-kube-api-access-ph66n\") pod \"5ef50367-92c0-4c69-9612-2a4829727c07\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") "
Jun 15 06:22:25.183585 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.183536    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef50367-92c0-4c69-9612-2a4829727c07-kube-rbac-proxy-sar-config\") pod \"5ef50367-92c0-4c69-9612-2a4829727c07\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") "
Jun 15 06:22:25.183585 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.183585    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef50367-92c0-4c69-9612-2a4829727c07-proxy-tls\") pod \"5ef50367-92c0-4c69-9612-2a4829727c07\" (UID: \"5ef50367-92c0-4c69-9612-2a4829727c07\") "
Jun 15 06:22:25.183981 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.183954    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef50367-92c0-4c69-9612-2a4829727c07-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ef50367-92c0-4c69-9612-2a4829727c07" (UID: "5ef50367-92c0-4c69-9612-2a4829727c07"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:22:25.185553 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.185522    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef50367-92c0-4c69-9612-2a4829727c07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ef50367-92c0-4c69-9612-2a4829727c07" (UID: "5ef50367-92c0-4c69-9612-2a4829727c07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:22:25.185655 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.185598    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef50367-92c0-4c69-9612-2a4829727c07-kube-api-access-ph66n" (OuterVolumeSpecName: "kube-api-access-ph66n") pod "5ef50367-92c0-4c69-9612-2a4829727c07" (UID: "5ef50367-92c0-4c69-9612-2a4829727c07"). InnerVolumeSpecName "kube-api-access-ph66n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:22:25.284639 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.284605    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph66n\" (UniqueName: \"kubernetes.io/projected/5ef50367-92c0-4c69-9612-2a4829727c07-kube-api-access-ph66n\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:22:25.284639 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.284632    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ef50367-92c0-4c69-9612-2a4829727c07-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:22:25.284639 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.284642    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ef50367-92c0-4c69-9612-2a4829727c07-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:22:25.505266 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.505218    2576 generic.go:356] "Generic (PLEG): container finished" podID="5ef50367-92c0-4c69-9612-2a4829727c07" containerID="8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a" exitCode=0
Jun 15 06:22:25.505663 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.505291    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"
Jun 15 06:22:25.505663 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.505302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" event={"ID":"5ef50367-92c0-4c69-9612-2a4829727c07","Type":"ContainerDied","Data":"8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a"}
Jun 15 06:22:25.505663 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.505362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt" event={"ID":"5ef50367-92c0-4c69-9612-2a4829727c07","Type":"ContainerDied","Data":"c053118dffe331bc1b34e73470c7e163656baa3a5f6b6d1a3283bb108920180b"}
Jun 15 06:22:25.505663 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.505390    2576 scope.go:117] "RemoveContainer" containerID="8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7"
Jun 15 06:22:25.505954 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.505916    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 15 06:22:25.514525 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.514507    2576 scope.go:117] "RemoveContainer" containerID="8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a"
Jun 15 06:22:25.521753 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.521737    2576 scope.go:117] "RemoveContainer" containerID="8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7"
Jun 15 06:22:25.522006 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:22:25.521988    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7\": container with ID starting with 8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7 not found: ID does not exist" containerID="8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7"
Jun 15 06:22:25.522046 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.522016    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7"} err="failed to get container status \"8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7\": rpc error: code = NotFound desc = could not find container \"8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7\": container with ID starting with 8eabc1c6c8f2087a57f2e82dfcd34fc8c675b8395f2ace8b0d32c9a93df6eea7 not found: ID does not exist"
Jun 15 06:22:25.522046 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.522034    2576 scope.go:117] "RemoveContainer" containerID="8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a"
Jun 15 06:22:25.522258 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:22:25.522229    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a\": container with ID starting with 8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a not found: ID does not exist" containerID="8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a"
Jun 15 06:22:25.522372 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.522353    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a"} err="failed to get container status \"8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a\": rpc error: code = NotFound desc = could not find container \"8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a\": container with ID starting with 8ef59bb0ed9eb6d39d374982ce170a9a3a1f11a916dec6af229c1d529bf2c63a not found: ID does not exist"
Jun 15 06:22:25.533352 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.533329    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"]
Jun 15 06:22:25.533352 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:25.533364    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44470-predictor-677b7d7455-tz4zt"]
Jun 15 06:22:27.533160 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:27.533111    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" path="/var/lib/kubelet/pods/5ef50367-92c0-4c69-9612-2a4829727c07/volumes"
Jun 15 06:22:30.510673 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:30.510638    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:22:30.511267 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:30.511217    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 15 06:22:36.213101 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.213063    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"]
Jun 15 06:22:36.213537 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.213321    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container" containerID="cri-o://b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909" gracePeriod=30
Jun 15 06:22:36.213537 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.213382    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kube-rbac-proxy" containerID="cri-o://b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518" gracePeriod=30
Jun 15 06:22:36.237793 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.237761    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"]
Jun 15 06:22:36.238288 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.238268    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kube-rbac-proxy"
Jun 15 06:22:36.238354 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.238293    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kube-rbac-proxy"
Jun 15 06:22:36.238354 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.238310    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container"
Jun 15 06:22:36.238354 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.238319    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container"
Jun 15 06:22:36.238450 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.238398    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kserve-container"
Jun 15 06:22:36.238450 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.238414    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef50367-92c0-4c69-9612-2a4829727c07" containerName="kube-rbac-proxy"
Jun 15 06:22:36.243152 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.243134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.245169 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.245149    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5edeb-predictor-serving-cert\""
Jun 15 06:22:36.245415 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.245392    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5edeb-kube-rbac-proxy-sar-config\""
Jun 15 06:22:36.248928 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.248903    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"]
Jun 15 06:22:36.264835 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.264803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1550a953-a029-4824-95cc-8bd781de6aff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.264933 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.264858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1550a953-a029-4824-95cc-8bd781de6aff-proxy-tls\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.264981 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.264939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtnl2\" (UniqueName: \"kubernetes.io/projected/1550a953-a029-4824-95cc-8bd781de6aff-kube-api-access-mtnl2\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.366169 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.366134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1550a953-a029-4824-95cc-8bd781de6aff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.366625 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.366192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1550a953-a029-4824-95cc-8bd781de6aff-proxy-tls\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.366625 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.366228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtnl2\" (UniqueName: \"kubernetes.io/projected/1550a953-a029-4824-95cc-8bd781de6aff-kube-api-access-mtnl2\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.366625 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:22:36.366395    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-serving-cert: secret "error-404-isvc-5edeb-predictor-serving-cert" not found
Jun 15 06:22:36.366625 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:22:36.366483    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1550a953-a029-4824-95cc-8bd781de6aff-proxy-tls podName:1550a953-a029-4824-95cc-8bd781de6aff nodeName:}" failed. No retries permitted until 2026-06-15 06:22:36.866462335 +0000 UTC m=+2089.774342844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1550a953-a029-4824-95cc-8bd781de6aff-proxy-tls") pod "error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" (UID: "1550a953-a029-4824-95cc-8bd781de6aff") : secret "error-404-isvc-5edeb-predictor-serving-cert" not found
Jun 15 06:22:36.366941 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.366920    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1550a953-a029-4824-95cc-8bd781de6aff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.374417 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.374389    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtnl2\" (UniqueName: \"kubernetes.io/projected/1550a953-a029-4824-95cc-8bd781de6aff-kube-api-access-mtnl2\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.551782 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.551688    2576 generic.go:356] "Generic (PLEG): container finished" podID="384b72c2-64bc-4ba4-baba-46973675935b" containerID="b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518" exitCode=2
Jun 15 06:22:36.551782 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.551751    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" event={"ID":"384b72c2-64bc-4ba4-baba-46973675935b","Type":"ContainerDied","Data":"b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518"}
Jun 15 06:22:36.871236 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.871132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1550a953-a029-4824-95cc-8bd781de6aff-proxy-tls\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:36.873609 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:36.873588    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1550a953-a029-4824-95cc-8bd781de6aff-proxy-tls\") pod \"error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") " pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:37.154345 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:37.154238    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:37.285788 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:37.285758    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"]
Jun 15 06:22:37.288191 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:22:37.288161    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1550a953_a029_4824_95cc_8bd781de6aff.slice/crio-270f27813674c6f70ac7ea6278308d92a19d567d52c38f9acbde8c4e6cfbda10 WatchSource:0}: Error finding container 270f27813674c6f70ac7ea6278308d92a19d567d52c38f9acbde8c4e6cfbda10: Status 404 returned error can't find the container with id 270f27813674c6f70ac7ea6278308d92a19d567d52c38f9acbde8c4e6cfbda10
Jun 15 06:22:37.557442 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:37.557399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" event={"ID":"1550a953-a029-4824-95cc-8bd781de6aff","Type":"ContainerStarted","Data":"3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6"}
Jun 15 06:22:37.557637 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:37.557474    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" event={"ID":"1550a953-a029-4824-95cc-8bd781de6aff","Type":"ContainerStarted","Data":"fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746"}
Jun 15 06:22:37.557637 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:37.557505    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" event={"ID":"1550a953-a029-4824-95cc-8bd781de6aff","Type":"ContainerStarted","Data":"270f27813674c6f70ac7ea6278308d92a19d567d52c38f9acbde8c4e6cfbda10"}
Jun 15 06:22:37.557637 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:37.557543    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:37.575218 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:37.575168    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" podStartSLOduration=1.575153393 podStartE2EDuration="1.575153393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:22:37.573382945 +0000 UTC m=+2090.481263460" watchObservedRunningTime="2026-06-15 06:22:37.575153393 +0000 UTC m=+2090.483033956"
Jun 15 06:22:38.561400 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:38.561369    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:38.562620 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:38.562594    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jun 15 06:22:39.281791 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.281744    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jun 15 06:22:39.565020 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.564992    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:22:39.565813 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.565788    2576 generic.go:356] "Generic (PLEG): container finished" podID="384b72c2-64bc-4ba4-baba-46973675935b" containerID="b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909" exitCode=0
Jun 15 06:22:39.565901 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.565855    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" event={"ID":"384b72c2-64bc-4ba4-baba-46973675935b","Type":"ContainerDied","Data":"b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909"}
Jun 15 06:22:39.565949 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.565909    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8" event={"ID":"384b72c2-64bc-4ba4-baba-46973675935b","Type":"ContainerDied","Data":"be6f8c99da2ce6c57c525b8f4d612aef7d62956188c697b8d90c3804eee1e20b"}
Jun 15 06:22:39.565984 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.565950    2576 scope.go:117] "RemoveContainer" containerID="b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518"
Jun 15 06:22:39.566170 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.566140    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jun 15 06:22:39.573771 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.573755    2576 scope.go:117] "RemoveContainer" containerID="b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909"
Jun 15 06:22:39.580971 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.580945    2576 scope.go:117] "RemoveContainer" containerID="b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518"
Jun 15 06:22:39.581217 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:22:39.581195    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518\": container with ID starting with b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518 not found: ID does not exist" containerID="b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518"
Jun 15 06:22:39.581299 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.581225    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518"} err="failed to get container status \"b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518\": rpc error: code = NotFound desc = could not find container \"b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518\": container with ID starting with b4fa4fc5f665953430852277ec47ec7246e47fff422e8e76c841be7c760d7518 not found: ID does not exist"
Jun 15 06:22:39.581299 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.581256    2576 scope.go:117] "RemoveContainer" containerID="b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909"
Jun 15 06:22:39.581485 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:22:39.581468    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909\": container with ID starting with b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909 not found: ID does not exist" containerID="b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909"
Jun 15 06:22:39.581526 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.581494    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909"} err="failed to get container status \"b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909\": rpc error: code = NotFound desc = could not find container \"b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909\": container with ID starting with b72c6186868db593a7fd4563506009f9c8602fb51abe9b238078e783ad17d909 not found: ID does not exist"
Jun 15 06:22:39.595097 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.595072    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfkpx\" (UniqueName: \"kubernetes.io/projected/384b72c2-64bc-4ba4-baba-46973675935b-kube-api-access-tfkpx\") pod \"384b72c2-64bc-4ba4-baba-46973675935b\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") "
Jun 15 06:22:39.595231 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.595131    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/384b72c2-64bc-4ba4-baba-46973675935b-kube-rbac-proxy-sar-config\") pod \"384b72c2-64bc-4ba4-baba-46973675935b\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") "
Jun 15 06:22:39.595231 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.595152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/384b72c2-64bc-4ba4-baba-46973675935b-proxy-tls\") pod \"384b72c2-64bc-4ba4-baba-46973675935b\" (UID: \"384b72c2-64bc-4ba4-baba-46973675935b\") "
Jun 15 06:22:39.595549 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.595512    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384b72c2-64bc-4ba4-baba-46973675935b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "384b72c2-64bc-4ba4-baba-46973675935b" (UID: "384b72c2-64bc-4ba4-baba-46973675935b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:22:39.597151 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.597127    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384b72c2-64bc-4ba4-baba-46973675935b-kube-api-access-tfkpx" (OuterVolumeSpecName: "kube-api-access-tfkpx") pod "384b72c2-64bc-4ba4-baba-46973675935b" (UID: "384b72c2-64bc-4ba4-baba-46973675935b"). InnerVolumeSpecName "kube-api-access-tfkpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:22:39.597297 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.597271    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384b72c2-64bc-4ba4-baba-46973675935b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "384b72c2-64bc-4ba4-baba-46973675935b" (UID: "384b72c2-64bc-4ba4-baba-46973675935b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:22:39.696481 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.696444    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfkpx\" (UniqueName: \"kubernetes.io/projected/384b72c2-64bc-4ba4-baba-46973675935b-kube-api-access-tfkpx\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:22:39.696481 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.696474    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/384b72c2-64bc-4ba4-baba-46973675935b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:22:39.696481 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:39.696485    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/384b72c2-64bc-4ba4-baba-46973675935b-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:22:40.511192 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:40.511152    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 15 06:22:40.570041 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:40.570015    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"
Jun 15 06:22:40.591128 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:40.591098    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"]
Jun 15 06:22:40.595197 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:40.595168    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5c5b6-predictor-6b459ddd67-6djb8"]
Jun 15 06:22:41.533026 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:41.532991    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="384b72c2-64bc-4ba4-baba-46973675935b" path="/var/lib/kubelet/pods/384b72c2-64bc-4ba4-baba-46973675935b/volumes"
Jun 15 06:22:44.569880 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:44.569850    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:22:44.570346 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:44.570319    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jun 15 06:22:47.648666 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:47.648639    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:22:47.654655 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:47.654634    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:22:50.511664 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:50.511621    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 15 06:22:54.571151 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:22:54.571107    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jun 15 06:23:00.512020 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:23:00.511979    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 15 06:23:04.570519 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:23:04.570475    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jun 15 06:23:10.512439 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:23:10.512405    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:23:14.570394 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:23:14.570354    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jun 15 06:23:24.570912 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:23:24.570882    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:27:47.672782 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:27:47.672753    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:27:47.679339 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:27:47.679318    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:31:51.063015 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:51.062970    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"]
Jun 15 06:31:51.065578 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:51.063288    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container" containerID="cri-o://fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746" gracePeriod=30
Jun 15 06:31:51.065578 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:51.063305    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kube-rbac-proxy" containerID="cri-o://3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6" gracePeriod=30
Jun 15 06:31:51.693350 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:51.693310    2576 generic.go:356] "Generic (PLEG): container finished" podID="1550a953-a029-4824-95cc-8bd781de6aff" containerID="3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6" exitCode=2
Jun 15 06:31:51.693523 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:51.693388    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" event={"ID":"1550a953-a029-4824-95cc-8bd781de6aff","Type":"ContainerDied","Data":"3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6"}
Jun 15 06:31:54.216448 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.216421    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:31:54.264389 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.264358    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1550a953-a029-4824-95cc-8bd781de6aff-kube-rbac-proxy-sar-config\") pod \"1550a953-a029-4824-95cc-8bd781de6aff\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") "
Jun 15 06:31:54.264581 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.264446    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1550a953-a029-4824-95cc-8bd781de6aff-proxy-tls\") pod \"1550a953-a029-4824-95cc-8bd781de6aff\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") "
Jun 15 06:31:54.264581 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.264505    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtnl2\" (UniqueName: \"kubernetes.io/projected/1550a953-a029-4824-95cc-8bd781de6aff-kube-api-access-mtnl2\") pod \"1550a953-a029-4824-95cc-8bd781de6aff\" (UID: \"1550a953-a029-4824-95cc-8bd781de6aff\") "
Jun 15 06:31:54.264761 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.264733    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1550a953-a029-4824-95cc-8bd781de6aff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1550a953-a029-4824-95cc-8bd781de6aff" (UID: "1550a953-a029-4824-95cc-8bd781de6aff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:31:54.266499 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.266465    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1550a953-a029-4824-95cc-8bd781de6aff-kube-api-access-mtnl2" (OuterVolumeSpecName: "kube-api-access-mtnl2") pod "1550a953-a029-4824-95cc-8bd781de6aff" (UID: "1550a953-a029-4824-95cc-8bd781de6aff"). InnerVolumeSpecName "kube-api-access-mtnl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:31:54.266499 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.266474    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1550a953-a029-4824-95cc-8bd781de6aff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1550a953-a029-4824-95cc-8bd781de6aff" (UID: "1550a953-a029-4824-95cc-8bd781de6aff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:31:54.365805 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.365717    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1550a953-a029-4824-95cc-8bd781de6aff-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:31:54.365805 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.365746    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtnl2\" (UniqueName: \"kubernetes.io/projected/1550a953-a029-4824-95cc-8bd781de6aff-kube-api-access-mtnl2\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:31:54.365805 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.365757    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1550a953-a029-4824-95cc-8bd781de6aff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:31:54.707011 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.706973    2576 generic.go:356] "Generic (PLEG): container finished" podID="1550a953-a029-4824-95cc-8bd781de6aff" containerID="fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746" exitCode=0
Jun 15 06:31:54.707176 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.707016    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"
Jun 15 06:31:54.707176 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.707049    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" event={"ID":"1550a953-a029-4824-95cc-8bd781de6aff","Type":"ContainerDied","Data":"fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746"}
Jun 15 06:31:54.707176 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.707113    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2" event={"ID":"1550a953-a029-4824-95cc-8bd781de6aff","Type":"ContainerDied","Data":"270f27813674c6f70ac7ea6278308d92a19d567d52c38f9acbde8c4e6cfbda10"}
Jun 15 06:31:54.707176 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.707145    2576 scope.go:117] "RemoveContainer" containerID="3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6"
Jun 15 06:31:54.716240 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.716215    2576 scope.go:117] "RemoveContainer" containerID="fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746"
Jun 15 06:31:54.723554 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.723536    2576 scope.go:117] "RemoveContainer" containerID="3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6"
Jun 15 06:31:54.723818 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:31:54.723801    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6\": container with ID starting with 3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6 not found: ID does not exist" containerID="3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6"
Jun 15 06:31:54.723877 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.723826    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6"} err="failed to get container status \"3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6\": rpc error: code = NotFound desc = could not find container \"3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6\": container with ID starting with 3aca08735a644aaa72cbd3c7bca0982db52b99ee2160c2c3baafb2a7ff6a48d6 not found: ID does not exist"
Jun 15 06:31:54.723877 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.723845    2576 scope.go:117] "RemoveContainer" containerID="fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746"
Jun 15 06:31:54.724065 ip-10-0-129-86 kubenswrapper[2576]: E0615 06:31:54.724045    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746\": container with ID starting with fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746 not found: ID does not exist" containerID="fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746"
Jun 15 06:31:54.724124 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.724078    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746"} err="failed to get container status \"fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746\": rpc error: code = NotFound desc = could not find container \"fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746\": container with ID starting with fdf789f6508c4ef564ead43f1cc5b4940e1c038d6067fa71dce0c0b47db93746 not found: ID does not exist"
Jun 15 06:31:54.729946 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.729916    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"]
Jun 15 06:31:54.735363 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:54.735343    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5edeb-predictor-66b854b5f9-5dfb2"]
Jun 15 06:31:55.533171 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:31:55.533139    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1550a953-a029-4824-95cc-8bd781de6aff" path="/var/lib/kubelet/pods/1550a953-a029-4824-95cc-8bd781de6aff/volumes"
Jun 15 06:32:47.701822 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:32:47.701733    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:32:47.705703 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:32:47.705681    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:37:47.725702 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:37:47.725672    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:37:47.736756 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:37:47.736729    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:39:41.349666 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:41.349622    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"]
Jun 15 06:39:41.350289 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:41.349858    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container" containerID="cri-o://c4a502135064ea2d7279f03217ff6f4ee7e59f94fe973234b5a788a79ee3a37e" gracePeriod=30
Jun 15 06:39:41.350289 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:41.349955    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kube-rbac-proxy" containerID="cri-o://9aa4d9727041fec18a51979607a7b0421287ee4779653bec747439ffaaa7dc53" gracePeriod=30
Jun 15 06:39:42.457667 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:42.457632    2576 generic.go:356] "Generic (PLEG): container finished" podID="5efa7977-0132-444b-8cd0-956adf38943c" containerID="9aa4d9727041fec18a51979607a7b0421287ee4779653bec747439ffaaa7dc53" exitCode=2
Jun 15 06:39:42.458108 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:42.457698    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" event={"ID":"5efa7977-0132-444b-8cd0-956adf38943c","Type":"ContainerDied","Data":"9aa4d9727041fec18a51979607a7b0421287ee4779653bec747439ffaaa7dc53"}
Jun 15 06:39:44.468083 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.468039    2576 generic.go:356] "Generic (PLEG): container finished" podID="5efa7977-0132-444b-8cd0-956adf38943c" containerID="c4a502135064ea2d7279f03217ff6f4ee7e59f94fe973234b5a788a79ee3a37e" exitCode=0
Jun 15 06:39:44.468568 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.468088    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" event={"ID":"5efa7977-0132-444b-8cd0-956adf38943c","Type":"ContainerDied","Data":"c4a502135064ea2d7279f03217ff6f4ee7e59f94fe973234b5a788a79ee3a37e"}
Jun 15 06:39:44.498352 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.498329    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:39:44.618557 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.618454    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5efa7977-0132-444b-8cd0-956adf38943c-kube-rbac-proxy-sar-config\") pod \"5efa7977-0132-444b-8cd0-956adf38943c\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") "
Jun 15 06:39:44.618557 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.618497    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5efa7977-0132-444b-8cd0-956adf38943c-proxy-tls\") pod \"5efa7977-0132-444b-8cd0-956adf38943c\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") "
Jun 15 06:39:44.618557 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.618551    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkggq\" (UniqueName: \"kubernetes.io/projected/5efa7977-0132-444b-8cd0-956adf38943c-kube-api-access-fkggq\") pod \"5efa7977-0132-444b-8cd0-956adf38943c\" (UID: \"5efa7977-0132-444b-8cd0-956adf38943c\") "
Jun 15 06:39:44.618901 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.618875    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efa7977-0132-444b-8cd0-956adf38943c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5efa7977-0132-444b-8cd0-956adf38943c" (UID: "5efa7977-0132-444b-8cd0-956adf38943c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:39:44.619097 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.619077    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5efa7977-0132-444b-8cd0-956adf38943c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:39:44.620669 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.620636    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5efa7977-0132-444b-8cd0-956adf38943c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5efa7977-0132-444b-8cd0-956adf38943c" (UID: "5efa7977-0132-444b-8cd0-956adf38943c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:39:44.620780 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.620716    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efa7977-0132-444b-8cd0-956adf38943c-kube-api-access-fkggq" (OuterVolumeSpecName: "kube-api-access-fkggq") pod "5efa7977-0132-444b-8cd0-956adf38943c" (UID: "5efa7977-0132-444b-8cd0-956adf38943c"). InnerVolumeSpecName "kube-api-access-fkggq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:39:44.720116 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.720087    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkggq\" (UniqueName: \"kubernetes.io/projected/5efa7977-0132-444b-8cd0-956adf38943c-kube-api-access-fkggq\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:39:44.720116 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:44.720113    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5efa7977-0132-444b-8cd0-956adf38943c-proxy-tls\") on node \"ip-10-0-129-86.ec2.internal\" DevicePath \"\""
Jun 15 06:39:45.473529 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:45.473499    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"
Jun 15 06:39:45.473529 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:45.473522    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k" event={"ID":"5efa7977-0132-444b-8cd0-956adf38943c","Type":"ContainerDied","Data":"326f8f4afdcd33b4e4cc07cc02c1067259ee2c2c13a39ad2bef8d0f3216c5291"}
Jun 15 06:39:45.474003 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:45.473584    2576 scope.go:117] "RemoveContainer" containerID="9aa4d9727041fec18a51979607a7b0421287ee4779653bec747439ffaaa7dc53"
Jun 15 06:39:45.481888 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:45.481870    2576 scope.go:117] "RemoveContainer" containerID="c4a502135064ea2d7279f03217ff6f4ee7e59f94fe973234b5a788a79ee3a37e"
Jun 15 06:39:45.494754 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:45.494724    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"]
Jun 15 06:39:45.498194 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:45.498171    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c800-predictor-54776894c5-6zr7k"]
Jun 15 06:39:45.532208 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:39:45.532174    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5efa7977-0132-444b-8cd0-956adf38943c" path="/var/lib/kubelet/pods/5efa7977-0132-444b-8cd0-956adf38943c/volumes"
Jun 15 06:40:13.644608 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644522    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8rxx/must-gather-lxkbb"]
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644823    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kube-rbac-proxy"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644834    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kube-rbac-proxy"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644845    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644850    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644859    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kube-rbac-proxy"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644864    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kube-rbac-proxy"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644877    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644882    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644889    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644894    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644903    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kube-rbac-proxy"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644908    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kube-rbac-proxy"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644953    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kserve-container"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644962    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kserve-container"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644969    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kserve-container"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644974    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="384b72c2-64bc-4ba4-baba-46973675935b" containerName="kube-rbac-proxy"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644979    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5efa7977-0132-444b-8cd0-956adf38943c" containerName="kube-rbac-proxy"
Jun 15 06:40:13.645052 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.644984    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1550a953-a029-4824-95cc-8bd781de6aff" containerName="kube-rbac-proxy"
Jun 15 06:40:13.647837 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.647820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rxx/must-gather-lxkbb"
Jun 15 06:40:13.650333 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.650311    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8rxx\"/\"openshift-service-ca.crt\""
Jun 15 06:40:13.651136 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.651118    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8rxx\"/\"kube-root-ca.crt\""
Jun 15 06:40:13.651232 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.651177    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8rxx\"/\"default-dockercfg-t4kgc\""
Jun 15 06:40:13.654986 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.654947    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rxx/must-gather-lxkbb"]
Jun 15 06:40:13.736692 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.736662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6bh\" (UniqueName: \"kubernetes.io/projected/024258ca-e664-4bb8-a5a2-6198dd98399d-kube-api-access-xf6bh\") pod \"must-gather-lxkbb\" (UID: \"024258ca-e664-4bb8-a5a2-6198dd98399d\") " pod="openshift-must-gather-j8rxx/must-gather-lxkbb"
Jun 15 06:40:13.736849 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.736699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/024258ca-e664-4bb8-a5a2-6198dd98399d-must-gather-output\") pod \"must-gather-lxkbb\" (UID: \"024258ca-e664-4bb8-a5a2-6198dd98399d\") " pod="openshift-must-gather-j8rxx/must-gather-lxkbb"
Jun 15 06:40:13.837672 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.837630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6bh\" (UniqueName: \"kubernetes.io/projected/024258ca-e664-4bb8-a5a2-6198dd98399d-kube-api-access-xf6bh\") pod \"must-gather-lxkbb\" (UID: \"024258ca-e664-4bb8-a5a2-6198dd98399d\") " pod="openshift-must-gather-j8rxx/must-gather-lxkbb"
Jun 15 06:40:13.837672 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.837674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/024258ca-e664-4bb8-a5a2-6198dd98399d-must-gather-output\") pod \"must-gather-lxkbb\" (UID: \"024258ca-e664-4bb8-a5a2-6198dd98399d\") " pod="openshift-must-gather-j8rxx/must-gather-lxkbb"
Jun 15 06:40:13.837990 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.837975    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/024258ca-e664-4bb8-a5a2-6198dd98399d-must-gather-output\") pod \"must-gather-lxkbb\" (UID: \"024258ca-e664-4bb8-a5a2-6198dd98399d\") " pod="openshift-must-gather-j8rxx/must-gather-lxkbb"
Jun 15 06:40:13.845632 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.845608    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6bh\" (UniqueName: \"kubernetes.io/projected/024258ca-e664-4bb8-a5a2-6198dd98399d-kube-api-access-xf6bh\") pod \"must-gather-lxkbb\" (UID: \"024258ca-e664-4bb8-a5a2-6198dd98399d\") " pod="openshift-must-gather-j8rxx/must-gather-lxkbb"
Jun 15 06:40:13.956938 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:13.956904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rxx/must-gather-lxkbb"
Jun 15 06:40:14.087717 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:14.087697    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rxx/must-gather-lxkbb"]
Jun 15 06:40:14.090325 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:40:14.090296    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod024258ca_e664_4bb8_a5a2_6198dd98399d.slice/crio-ec3a1670601595ae1b97edb68ac9d6264295e3586b37e80376a1c877a6d0aa46 WatchSource:0}: Error finding container ec3a1670601595ae1b97edb68ac9d6264295e3586b37e80376a1c877a6d0aa46: Status 404 returned error can't find the container with id ec3a1670601595ae1b97edb68ac9d6264295e3586b37e80376a1c877a6d0aa46
Jun 15 06:40:14.092173 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:14.092159    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 06:40:14.581797 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:14.581759    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rxx/must-gather-lxkbb" event={"ID":"024258ca-e664-4bb8-a5a2-6198dd98399d","Type":"ContainerStarted","Data":"ec3a1670601595ae1b97edb68ac9d6264295e3586b37e80376a1c877a6d0aa46"}
Jun 15 06:40:15.587561 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:15.587524    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rxx/must-gather-lxkbb" event={"ID":"024258ca-e664-4bb8-a5a2-6198dd98399d","Type":"ContainerStarted","Data":"3010c1ba2d938129ec25d9b674f890f3696558c702edf44b8639518145f8bebe"}
Jun 15 06:40:15.587561 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:15.587566    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rxx/must-gather-lxkbb" event={"ID":"024258ca-e664-4bb8-a5a2-6198dd98399d","Type":"ContainerStarted","Data":"285c0d2298dfca00fcd2b38ab01942fe7caa346b587c6c263a57496136ba63a9"}
Jun 15 06:40:15.604043 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:15.603992    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8rxx/must-gather-lxkbb" podStartSLOduration=1.847978481 podStartE2EDuration="2.603975369s" totalImagesPullingTime="755.996888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:40:15.60340631 +0000 UTC m=+3148.511286835" watchObservedRunningTime="2026-06-15 06:40:15.603975369 +0000 UTC m=+3148.511855881"
Jun 15 06:40:16.332491 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:16.332456    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2czf6_6a0fdb7a-b248-4f55-80eb-f9cc18e0ec53/global-pull-secret-syncer/0.log"
Jun 15 06:40:16.522888 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:16.522855    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mtz4k_19d542ed-18cc-46f9-bf7a-959f70dd7077/konnectivity-agent/0.log"
Jun 15 06:40:16.546665 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:16.546626    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-86.ec2.internal_cbc9805fb56ac37d1dbd6954f1bb4020/haproxy/0.log"
Jun 15 06:40:21.156471 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.156434    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-678bc549d7-wxn9c_9062eb8d-3809-445d-a781-df948d8a9e44/kube-state-metrics/0.log"
Jun 15 06:40:21.191107 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.191066    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-678bc549d7-wxn9c_9062eb8d-3809-445d-a781-df948d8a9e44/kube-rbac-proxy-main/0.log"
Jun 15 06:40:21.218810 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.218779    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-678bc549d7-wxn9c_9062eb8d-3809-445d-a781-df948d8a9e44/kube-rbac-proxy-self/0.log"
Jun 15 06:40:21.279694 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.279657    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-589576dfc7-jwrw9_fabf544d-8e04-4f91-aaae-da7bc43d9ec3/monitoring-plugin/0.log"
Jun 15 06:40:21.322920 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.322888    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-td7mk_ee61ea6a-ac85-4bb0-93c7-9a0cad05f775/node-exporter/0.log"
Jun 15 06:40:21.350108 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.350072    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-td7mk_ee61ea6a-ac85-4bb0-93c7-9a0cad05f775/kube-rbac-proxy/0.log"
Jun 15 06:40:21.376831 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.376796    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-td7mk_ee61ea6a-ac85-4bb0-93c7-9a0cad05f775/init-textfile/0.log"
Jun 15 06:40:21.571867 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.571834    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-g2kmc_d5c32636-2166-400e-b6bb-ca3f6c6de016/kube-rbac-proxy-main/0.log"
Jun 15 06:40:21.599271 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.599229    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-g2kmc_d5c32636-2166-400e-b6bb-ca3f6c6de016/kube-rbac-proxy-self/0.log"
Jun 15 06:40:21.622152 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:21.622115    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-g2kmc_d5c32636-2166-400e-b6bb-ca3f6c6de016/openshift-state-metrics/0.log"
Jun 15 06:40:22.197302 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:22.197267    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d655695f-cdz6t_b99a1226-c323-4db5-a0e7-8193b0b20e30/thanos-query/0.log"
Jun 15 06:40:22.221134 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:22.221103    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d655695f-cdz6t_b99a1226-c323-4db5-a0e7-8193b0b20e30/kube-rbac-proxy-web/0.log"
Jun 15 06:40:22.245935 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:22.245899    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d655695f-cdz6t_b99a1226-c323-4db5-a0e7-8193b0b20e30/kube-rbac-proxy/0.log"
Jun 15 06:40:22.273004 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:22.272974    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d655695f-cdz6t_b99a1226-c323-4db5-a0e7-8193b0b20e30/prom-label-proxy/0.log"
Jun 15 06:40:22.298426 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:22.298337    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d655695f-cdz6t_b99a1226-c323-4db5-a0e7-8193b0b20e30/kube-rbac-proxy-rules/0.log"
Jun 15 06:40:22.326369 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:22.326278    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d655695f-cdz6t_b99a1226-c323-4db5-a0e7-8193b0b20e30/kube-rbac-proxy-metrics/0.log"
Jun 15 06:40:24.377599 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.377558    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-694c955b9-vqxmq_627f8f86-bf57-4a37-92d5-8933d5eafa31/download-server/0.log"
Jun 15 06:40:24.386108 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.386075    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"]
Jun 15 06:40:24.391996 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.391964    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.398877 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.398840    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"]
Jun 15 06:40:24.431151 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.431108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-podres\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.431462 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.431436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-lib-modules\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.431656 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.431639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-sys\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.431775 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.431762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-proc\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.431871 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.431858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttxw\" (UniqueName: \"kubernetes.io/projected/275d894f-4274-47c6-b6a1-9cbe91a0392a-kube-api-access-7ttxw\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.532854 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.532818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-podres\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.532854 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.532861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-lib-modules\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.533047 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.532978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-sys\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.533047 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.532998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-proc\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.533047 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.533016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttxw\" (UniqueName: \"kubernetes.io/projected/275d894f-4274-47c6-b6a1-9cbe91a0392a-kube-api-access-7ttxw\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.533047 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.533018    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-podres\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.533177 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.533071    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-lib-modules\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.533177 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.533116    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-proc\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.533177 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.533122    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/275d894f-4274-47c6-b6a1-9cbe91a0392a-sys\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.542084 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.542050    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttxw\" (UniqueName: \"kubernetes.io/projected/275d894f-4274-47c6-b6a1-9cbe91a0392a-kube-api-access-7ttxw\") pod \"perf-node-gather-daemonset-vmr4r\" (UID: \"275d894f-4274-47c6-b6a1-9cbe91a0392a\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.707292 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.707236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:24.883322 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:24.883183    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"]
Jun 15 06:40:24.888411 ip-10-0-129-86 kubenswrapper[2576]: W0615 06:40:24.888371    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod275d894f_4274_47c6_b6a1_9cbe91a0392a.slice/crio-cda85853c8abf7d4373656233cd67c3c393c5437baf4acccbea007e19219bfab WatchSource:0}: Error finding container cda85853c8abf7d4373656233cd67c3c393c5437baf4acccbea007e19219bfab: Status 404 returned error can't find the container with id cda85853c8abf7d4373656233cd67c3c393c5437baf4acccbea007e19219bfab
Jun 15 06:40:25.637788 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:25.637744    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r" event={"ID":"275d894f-4274-47c6-b6a1-9cbe91a0392a","Type":"ContainerStarted","Data":"3c51376f833a50b23988e77b6cccfed5579f5d3dd22f05f6f4146f0efdbee764"}
Jun 15 06:40:25.638354 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:25.637815    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r" event={"ID":"275d894f-4274-47c6-b6a1-9cbe91a0392a","Type":"ContainerStarted","Data":"cda85853c8abf7d4373656233cd67c3c393c5437baf4acccbea007e19219bfab"}
Jun 15 06:40:25.638354 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:25.637846    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:25.655596 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:25.654700    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r" podStartSLOduration=1.6546840459999999 podStartE2EDuration="1.654684046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 06:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:40:25.653685402 +0000 UTC m=+3158.561565916" watchObservedRunningTime="2026-06-15 06:40:25.654684046 +0000 UTC m=+3158.562564558"
Jun 15 06:40:25.921126 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:25.921031    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kn75r_029b7613-6544-4c5b-aef8-3041d968a75f/dns/0.log"
Jun 15 06:40:25.944383 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:25.944352    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kn75r_029b7613-6544-4c5b-aef8-3041d968a75f/kube-rbac-proxy/0.log"
Jun 15 06:40:26.044854 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:26.044822    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pspvw_52ce0a05-4c82-4b03-8386-6ebdb0a6bae5/dns-node-resolver/0.log"
Jun 15 06:40:26.592555 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:26.592527    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5k6jr_ecd20280-4ff2-42de-b4e2-85cdf88de891/node-ca/0.log"
Jun 15 06:40:27.879282 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:27.879238    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hslrg_1e3e29ab-7a28-4dd1-8d55-a3115197a16c/serve-healthcheck-canary/0.log"
Jun 15 06:40:28.401868 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:28.401840    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sg2hl_2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f/kube-rbac-proxy/0.log"
Jun 15 06:40:28.421951 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:28.421911    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sg2hl_2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f/exporter/0.log"
Jun 15 06:40:28.442891 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:28.442863    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sg2hl_2d58ceb3-e0b2-41e8-abdc-e0e6c6ce015f/extractor/0.log"
Jun 15 06:40:30.418366 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:30.418333    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-79wdh_a2fef3c5-e630-4abf-8e4e-9ad07ef31813/manager/0.log"
Jun 15 06:40:30.614657 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:30.614618    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-prlg8_71b5b062-e119-49cc-a3a3-f74f6d3072cf/manager/0.log"
Jun 15 06:40:31.653184 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:31.653151    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-vmr4r"
Jun 15 06:40:36.299008 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.298977    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2s9g_3a3e4383-5637-45fd-b49d-9d77f7573feb/kube-multus-additional-cni-plugins/0.log"
Jun 15 06:40:36.322993 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.322965    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2s9g_3a3e4383-5637-45fd-b49d-9d77f7573feb/egress-router-binary-copy/0.log"
Jun 15 06:40:36.344410 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.344386    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2s9g_3a3e4383-5637-45fd-b49d-9d77f7573feb/cni-plugins/0.log"
Jun 15 06:40:36.365690 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.365666    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2s9g_3a3e4383-5637-45fd-b49d-9d77f7573feb/bond-cni-plugin/0.log"
Jun 15 06:40:36.386694 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.386666    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2s9g_3a3e4383-5637-45fd-b49d-9d77f7573feb/routeoverride-cni/0.log"
Jun 15 06:40:36.407167 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.407138    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2s9g_3a3e4383-5637-45fd-b49d-9d77f7573feb/whereabouts-cni-bincopy/0.log"
Jun 15 06:40:36.427574 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.427547    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f2s9g_3a3e4383-5637-45fd-b49d-9d77f7573feb/whereabouts-cni/0.log"
Jun 15 06:40:36.669353 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.669325    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xf8tc_8d79a647-5223-4980-8d70-40168d1c14fb/kube-multus/0.log"
Jun 15 06:40:36.690826 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.690703    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cfxz2_c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2/network-metrics-daemon/0.log"
Jun 15 06:40:36.714439 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:36.714399    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cfxz2_c1b82b73-9ed8-4501-9cb0-b82bb6c21cf2/kube-rbac-proxy/0.log"
Jun 15 06:40:37.602765 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:37.602737    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-controller/0.log"
Jun 15 06:40:37.618420 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:37.618395    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/0.log"
Jun 15 06:40:37.618420 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:37.618416    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovn-acl-logging/1.log"
Jun 15 06:40:37.644017 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:37.643989    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/kube-rbac-proxy-node/0.log"
Jun 15 06:40:37.670343 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:37.670310    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/kube-rbac-proxy-ovn-metrics/0.log"
Jun 15 06:40:37.686718 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:37.686687    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/northd/0.log"
Jun 15 06:40:37.707801 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:37.707766    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/nbdb/0.log"
Jun 15 06:40:37.734352 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:37.734323    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/sbdb/0.log"
Jun 15 06:40:37.798632 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:37.798598    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb7k9_52951973-c105-489a-9d13-b213aff03d3d/ovnkube-controller/0.log"
Jun 15 06:40:39.281549 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:39.281522    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wsmkh_e586e4d3-72be-41a1-8b52-48301c1b36f4/network-check-target-container/0.log"
Jun 15 06:40:40.347795 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:40.347758    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-zk5nh_bb7bb954-4346-4175-8721-5ffbab5ef3d8/iptables-alerter/0.log"
Jun 15 06:40:41.087206 ip-10-0-129-86 kubenswrapper[2576]: I0615 06:40:41.087177    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-sckf9_1739a3c1-1f89-4206-b475-4c4b7374c36d/tuned/0.log"