Jul 29 10:28:36.367041 ip-10-0-129-138 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 10:28:36.770836 ip-10-0-129-138 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:36.770836 ip-10-0-129-138 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 10:28:36.770836 ip-10-0-129-138 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:36.770836 ip-10-0-129-138 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 10:28:36.770836 ip-10-0-129-138 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:36.773481 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.773385    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 10:28:36.777735 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777715    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:36.777735 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777733    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:36.777735 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777738    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777741    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777745    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777748    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777751    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777754    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777757    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777760    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777763    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777765    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777769    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777773    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777776    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777779    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777782    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777785    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777787    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777791    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777794    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:36.777843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777797    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777799    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777802    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777805    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777808    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777811    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777814    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777816    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777819    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777824    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777828    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777832    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777834    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777837    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777840    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777843    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777846    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777849    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777851    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777854    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:36.778308 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777856    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777859    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777862    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777865    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777868    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777871    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777874    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777877    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777879    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777882    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777885    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777887    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777890    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777894    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777898    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777904    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777908    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777912    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777916    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777920    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:36.778848 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777923    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777927    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777931    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777936    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777940    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777947    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777953    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777957    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777961    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777966    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777969    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777974    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777978    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777984    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777988    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777992    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.777996    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778000    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778004    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778007    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:36.779387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778012    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778018    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778022    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778026    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778030    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778034    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778038    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778042    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778046    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778050    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778053    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778058    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778062    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778066    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778070    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778074    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778080    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:36.779892 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.778085    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780055    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780066    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780070    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780074    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780077    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780080    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780083    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780085    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780090    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780093    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780095    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780098    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780101    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780104    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780106    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780111    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780114    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780117    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:36.780304 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780121    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780125    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780128    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780131    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780134    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780137    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780140    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780143    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780146    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780148    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780151    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780154    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780158    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780160    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780162    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780166    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780169    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780172    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780174    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780177    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:36.780794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780179    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780182    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780184    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780187    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780190    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780193    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780195    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780198    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780200    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780203    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780205    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780208    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780211    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780213    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780216    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780218    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780220    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780223    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780226    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780228    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:36.781284 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780231    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780234    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780238    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780241    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780243    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780246    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780248    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780251    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780254    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780256    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780259    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780261    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780264    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780267    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780270    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780274    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780278    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780281    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780284    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780287    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:36.781786 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780289    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780292    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780295    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780298    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780300    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780303    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780306    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780309    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780312    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780315    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780318    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780321    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780323    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780326    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780328    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780331    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780333    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780336    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780338    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780341    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:36.782285 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780344    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780776    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780781    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780784    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780787    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780790    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780793    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780796    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780800    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780804    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780807    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780809    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780812    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780815    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780817    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780821    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780824    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780827    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780830    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:36.782794 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780832    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780835    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780838    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780842    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780845    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780848    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780851    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780854    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780857    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780860    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780863    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780866    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780869    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780871    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780874    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780877    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780879    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780882    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780884    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:36.783259 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780887    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780890    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780893    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780897    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780900    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780903    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780905    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780908    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780910    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780913    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780915    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780918    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780920    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780924    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780926    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780929    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780934    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780937    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780940    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780943    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:36.783771 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780946    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780948    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780951    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780955    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780958    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780961    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780963    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780966    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780968    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780971    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780973    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780976    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780979    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780981    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780984    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780986    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780989    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780992    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780994    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:36.784300 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780997    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.780999    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781002    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781004    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781007    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781009    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781012    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781014    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781016    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781019    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781021    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781024    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781026    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781029    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781031    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781034    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781036    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781039    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781042    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781044    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:36.784793 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781047    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781049    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781052    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781139    2572 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781147    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781153    2572 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781158    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781163    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781166    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781170    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781176    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781179    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781182    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781186    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781190    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781193    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781196    2572 flags.go:64] FLAG: --cgroup-root=""
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781199    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781203    2572 flags.go:64] FLAG: --client-ca-file=""
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781205    2572 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781208    2572 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781212    2572 flags.go:64] FLAG: --cluster-domain=""
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781215    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781219    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 10:28:36.785273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781222    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781225    2572 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781230    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781233    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781236    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781240    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781243    2572 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781246    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781249    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781252    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781255    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781259    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781262    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781265    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781269    2572 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781272    2572 flags.go:64] FLAG: --enable-server="true"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781275    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781279    2572 flags.go:64] FLAG: --event-burst="100"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781283    2572 flags.go:64] FLAG: --event-qps="50"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781286    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781290    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781293    2572 flags.go:64] FLAG: --eviction-hard=""
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781297    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781300    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781303    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 10:28:36.785896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781306    2572 flags.go:64] FLAG: --eviction-soft=""
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781309    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781312    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781315    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781318    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781321    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781324    2572 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781327    2572 flags.go:64] FLAG: --feature-gates=""
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781331    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781335    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781338    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781341    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781344    2572 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781347    2572 flags.go:64] FLAG: --help="false"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781350    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-129-138.ec2.internal"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781353    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781357    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781360    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781363    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781366    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781369    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781372    2572 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781375    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 10:28:36.787047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781378    2572 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781381    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781384    2572 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781387    2572 flags.go:64] FLAG: --kube-reserved=""
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781393    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781396    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781400    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781403    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781406    2572 flags.go:64] FLAG: --lock-file=""
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781409    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781414    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781417    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781423    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781426    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781429    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781432    2572 flags.go:64] FLAG: --logging-format="text"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781435    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781438    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781443    2572 flags.go:64] FLAG: --manifest-url=""
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781446    2572 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781451    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781454    2572 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781458    2572 flags.go:64] FLAG: --max-pods="110"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781461    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781465    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 10:28:36.788024 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781468    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781471    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781474    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781477    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781480    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781493    2572 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781496    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781500    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781503    2572 flags.go:64] FLAG: --pod-cidr=""
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781506    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781512    2572 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781515    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781519    2572 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781522    2572 flags.go:64] FLAG: --port="10250"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781525    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781528    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f581c5f47e66008e"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781531    2572 flags.go:64] FLAG: --qos-reserved=""
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781534    2572 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781539    2572 flags.go:64] FLAG: --register-node="true"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781542    2572 flags.go:64] FLAG: --register-with-taints=""
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781546    2572 flags.go:64] FLAG: --registry-burst="10"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781549    2572 flags.go:64] FLAG: --registry-qps="5"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781552    2572 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781555    2572 flags.go:64] FLAG: --reserved-memory=""
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781559    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 10:28:36.789002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781564    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781567    2572 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781570    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781572    2572 flags.go:64] FLAG: --runonce="false"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781576    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781579    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781582    2572 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781585    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781588    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781591    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781594    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781597    2572 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781600    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781603    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781606    2572 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781609    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781612    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781616    2572 flags.go:64] FLAG: --system-cgroups=""
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781618    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781624    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781627    2572 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781630    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781634    2572 flags.go:64] FLAG: --tls-min-version=""
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781637    2572 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781639    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 10:28:36.790170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781643    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781646    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781649    2572 flags.go:64] FLAG: --v="2"
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781653    2572 flags.go:64] FLAG: --version="false"
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781657    2572 flags.go:64] FLAG: --vmodule=""
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781661    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.781664    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781809    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781814    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781817    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781821    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781824    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781826    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781829    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781832    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781835    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781838    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781841    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781843    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781846    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781848    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:36.791295 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781851    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781854    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781857    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781860    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781862    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781865    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781868    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781871    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781873    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781876    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781879    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781883    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781886    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781888    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781891    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781894    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781896    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781899    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781904    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781908    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:36.792150 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781911    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781914    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781917    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781919    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781922    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781925    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781927    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781930    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781932    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781935    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781937    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781940    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781943    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781945    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781948    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781950    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781953    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781955    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781958    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:36.792974 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781961    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781964    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781966    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781969    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781973    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781976    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781979    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781982    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781985    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781987    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781990    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781994    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781997    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.781999    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782002    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782004    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782007    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782009    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782012    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:36.793452 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782015    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782017    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782020    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782022    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782025    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782028    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782031    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782033    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782036    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782038    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782041    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782043    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782046    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782048    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782051    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782054    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782057    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782061    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782063    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782067    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:36.794210 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782071    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:36.794706 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782075    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:36.794706 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782078    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:36.794706 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782080    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:36.794706 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782084    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:36.794706 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782086    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:36.794706 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.782089    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:36.794706 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.782818    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:36.794706 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.794255    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 10:28:36.794706 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.794277    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.794830    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.794842    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794908    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794914    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794918    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794922    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794925    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794927    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794931    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794934    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794937    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794941    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:36.794939 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794944    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794947    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794950    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794952    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794955    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794958    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794961    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794964    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794966    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794969    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794972    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794975    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794978    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794981    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794983    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794986    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794989    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794991    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794993    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794996    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:36.795258 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.794999    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795001    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795004    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795006    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795009    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795012    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795014    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795017    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795019    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795022    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795025    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795027    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795030    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795032    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795035    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795038    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795041    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795044    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795047    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795050    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:36.795769 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795052    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795055    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795058    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795060    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795063    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795066    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795070    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795074    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795077    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795080    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795083    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795086    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795088    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795091    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795094    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795098    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795101    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795104    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795106    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:36.796261 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795109    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795111    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795114    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795116    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795120    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795123    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795125    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795129    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795132    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795135    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795139    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795142    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795145    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795148    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795150    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795153    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795156    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795160    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795163    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:36.796755 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795167    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795170    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795172    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795175    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795177    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795180    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795183    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795186    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795188    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795191    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795193    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.795198    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795319    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795325    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795328    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:36.797214 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795331    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795334    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795345    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795349    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795352    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795356    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795360    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795363    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795366    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795370    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795373    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795376    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795378    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795382    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795386    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795389    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795391    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795394    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795397    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:36.797575 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795399    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795402    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795405    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795407    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795410    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795413    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795415    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795418    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795420    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795423    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795425    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795428    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795430    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795433    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795436    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795438    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795442    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795446    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795450    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:36.798071 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795454    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795458    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795461    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795464    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795467    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795470    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795473    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795475    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795478    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795481    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795483    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795486    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795488    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795491    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795494    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795496    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795499    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795501    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795504    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795506    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:36.798539 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795509    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795512    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795514    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795517    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795519    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795523    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795525    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795528    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795530    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795533    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795536    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795538    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795541    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795543    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795546    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795549    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795551    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795554    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795557    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795559    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:36.799043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795562    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795564    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795567    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795570    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795572    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795575    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795577    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795580    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795583    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795612    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795616    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795619    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795622    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795625    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795627    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795630    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795633    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:36.799548 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:36.795636    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:36.800077 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.795641    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:36.800077 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.795796    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 10:28:36.800077 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.797886    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 10:28:36.800077 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.799197    2572 server.go:1019] "Starting client certificate rotation"
Jul 29 10:28:36.800077 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.799319    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:36.800077 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.799380    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:36.801719 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.801683    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:36.803054 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.803026    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:36.810975 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.810951    2572 log.go:25] "Validated CRI v1 runtime API"
Jul 29 10:28:36.813419 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.813398    2572 log.go:25] "Validated CRI v1 image API"
Jul 29 10:28:36.814448 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.814432    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 10:28:36.821142 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.821111    2572 fs.go:135] Filesystem UUIDs: map[577af9b8-e499-4322-b9d9-a1a57e15fa94:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 af313da4-9ddc-4196-a29d-1478e1d450d4:/dev/nvme0n1p4]
Jul 29 10:28:36.821142 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.821135    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 10:28:36.826329 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.826215    2572 manager.go:216] Machine: {Timestamp:2026-07-29 10:28:36.824831596 +0000 UTC m=+0.337308805 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099706 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a6908f8c14b252b91656c702e31ba SystemUUID:ec2a6908-f8c1-4b25-2b91-656c702e31ba BootID:9e2d00b9-fe20-4ccf-8908-57c7d510d72c Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:b5:d7:a1:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:b5:d7:a1:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:25:c5:ab:6b:9b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 10:28:36.826329 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.826324    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 10:28:36.826465 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.826450    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 10:28:36.828051 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.828017    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 10:28:36.828199 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.828057    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 10:28:36.828735 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.828724    2572 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 10:28:36.828766 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.828739    2572 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 10:28:36.828766 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.828746    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:36.829422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.829412    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:36.830198 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.830188    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 10:28:36.832452 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.832437    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:36.832592 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.832582    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 10:28:36.832623 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.832619    2572 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 10:28:36.832650 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.832634    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 10:28:36.832650 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.832649    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 10:28:36.832732 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.832659    2572 kubelet.go:398] "Adding apiserver pod source"
Jul 29 10:28:36.832732 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.832669    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 10:28:36.833069 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.833043    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:36.833069 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.833071    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:36.835286 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.835152    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:36.836971 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.836942    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 10:28:36.839159 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.839140    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 10:28:36.839246 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.839173    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 10:28:36.841316 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841301    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 10:28:36.841383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841325    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 10:28:36.841383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841332    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 10:28:36.841383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841338    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 10:28:36.841383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841347    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 10:28:36.841383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841353    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 10:28:36.841383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841361    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 10:28:36.841383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841367    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 10:28:36.841383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841374    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 10:28:36.841383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841380    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 10:28:36.841621 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841389    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 10:28:36.841621 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.841395    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 10:28:36.842128 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.842116    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 10:28:36.842185 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.842130    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 10:28:36.843593 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.843561    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 10:28:36.845565 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.845549    2572 server.go:1268] "Started kubelet"
Jul 29 10:28:36.845669 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.845637    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 10:28:36.845736 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.845637    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 10:28:36.845736 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.845726    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 10:28:36.846548 ip-10-0-129-138 systemd[1]: Started Kubernetes Kubelet.
Jul 29 10:28:36.847566 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.847545    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 10:28:36.848122 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.848108    2572 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 10:28:36.852091 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.852069    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 10:28:36.853882 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.853840    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 10:28:36.853882 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.853867    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 10:28:36.854904 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.853810    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-138.ec2.internal.18c6bbea32a6ccbe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-138.ec2.internal,UID:ip-10-0-129-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-138.ec2.internal,},FirstTimestamp:2026-07-29 10:28:36.845513918 +0000 UTC m=+0.357991461,LastTimestamp:2026-07-29 10:28:36.845513918 +0000 UTC m=+0.357991461,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-138.ec2.internal,}"
Jul 29 10:28:36.855893 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.855300    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 10:28:36.855893 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.855320    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:36.855893 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.855420    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:36.855893 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.855853    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 10:28:36.855893 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.855856    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 10:28:36.855893 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.855877    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 10:28:36.856213 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.855987    2572 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 10:28:36.856213 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.855993    2572 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 10:28:36.856597 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.856578    2572 factory.go:55] Registering systemd factory
Jul 29 10:28:36.856597 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.856599    2572 factory.go:223] Registration of the systemd container factory successfully
Jul 29 10:28:36.856951 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.856931    2572 factory.go:153] Registering CRI-O factory
Jul 29 10:28:36.856951 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.856948    2572 factory.go:223] Registration of the crio container factory successfully
Jul 29 10:28:36.857133 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.857003    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 10:28:36.857133 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.857025    2572 factory.go:103] Registering Raw factory
Jul 29 10:28:36.857133 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.857042    2572 manager.go:1196] Started watching for new ooms in manager
Jul 29 10:28:36.857539 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.857523    2572 manager.go:349] Starting recovery of all containers
Jul 29 10:28:36.859829 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.859805    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5k2vf"
Jul 29 10:28:36.860063 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.860041    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 10:28:36.860129 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.860040    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 10:28:36.866645 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.866614    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5k2vf"
Jul 29 10:28:36.871237 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.871211    2572 manager.go:354] Recovery completed
Jul 29 10:28:36.877867 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.877842    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:36.879767 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.879743    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:36.879857 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.879785    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:36.879857 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.879800    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:36.880426 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.880411    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:36.880524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.880410    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 10:28:36.880524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.880477    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 10:28:36.880524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.880523    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:36.881222 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.881197    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:36.881320 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.881227    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:36.881320 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.881237    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:36.882352 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.882337    2572 policy_none.go:49] "None policy: Start"
Jul 29 10:28:36.882352 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.882353    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 10:28:36.882487 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.882364    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 10:28:36.883405 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.883389    2572 policy_none.go:47] "Start"
Jul 29 10:28:36.924459 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.924444    2572 manager.go:337] "Starting Device Plugin manager"
Jul 29 10:28:36.924564 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.924481    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 10:28:36.924564 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.924495    2572 server.go:87] "Starting device plugin registration server"
Jul 29 10:28:36.924817 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.924803    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 10:28:36.924878 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.924817    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 10:28:36.924975 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.924958    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 10:28:36.925422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.925052    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 10:28:36.925422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:36.925064    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 10:28:36.925576 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.925538    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 10:28:36.925624 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:36.925587    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:37.018612 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.018447    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 10:28:37.019797 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.019776    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 10:28:37.019890 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.019830    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 10:28:37.019890 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.019859    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 10:28:37.019965 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:37.019902    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 10:28:37.022358 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.022303    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:37.025431 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.025413    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:37.026304 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.026285    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:37.026392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.026318    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:37.026392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.026342    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:37.026392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.026365    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.032376 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.032360    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.032423 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:37.032384    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-138.ec2.internal\": node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:37.060349 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:37.060322    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:37.120655 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.120623    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal"]
Jul 29 10:28:37.120745 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.120722    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:37.122410 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.122392    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:37.122505 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.122423    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:37.122505 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.122437    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:37.123750 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.123736    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:37.123861 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.123848    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.123897 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.123877    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:37.125932 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.125916    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:37.126031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.125941    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:37.126031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.125951    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:37.126031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.125965    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:37.126031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.125984    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:37.126031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.125993    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:37.127248 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.127234    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.127299 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.127264    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:37.130626 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.130609    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:37.130761 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.130644    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:37.130761 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.130659    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:37.142367 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:37.142344    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-138.ec2.internal\" not found" node="ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.148064 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:37.148043    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-138.ec2.internal\" not found" node="ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.158606 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.158580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da30ffe3ac8c128d369bee685e85d0a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal\" (UID: \"da30ffe3ac8c128d369bee685e85d0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.158761 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.158612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da30ffe3ac8c128d369bee685e85d0a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal\" (UID: \"da30ffe3ac8c128d369bee685e85d0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.158761 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.158634    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3e4af55a69b4ca5f48ea6f06887aafd-config\") pod \"kube-apiserver-proxy-ip-10-0-129-138.ec2.internal\" (UID: \"a3e4af55a69b4ca5f48ea6f06887aafd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.160705 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:37.160665    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:37.259170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.259138    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da30ffe3ac8c128d369bee685e85d0a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal\" (UID: \"da30ffe3ac8c128d369bee685e85d0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.259323 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.259187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3e4af55a69b4ca5f48ea6f06887aafd-config\") pod \"kube-apiserver-proxy-ip-10-0-129-138.ec2.internal\" (UID: \"a3e4af55a69b4ca5f48ea6f06887aafd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.259323 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.259206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da30ffe3ac8c128d369bee685e85d0a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal\" (UID: \"da30ffe3ac8c128d369bee685e85d0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.259323 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.259143    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da30ffe3ac8c128d369bee685e85d0a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal\" (UID: \"da30ffe3ac8c128d369bee685e85d0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.259323 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.259247    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3e4af55a69b4ca5f48ea6f06887aafd-config\") pod \"kube-apiserver-proxy-ip-10-0-129-138.ec2.internal\" (UID: \"a3e4af55a69b4ca5f48ea6f06887aafd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.259323 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:37.259234    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da30ffe3ac8c128d369bee685e85d0a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal\" (UID: \"da30ffe3ac8c128d369bee685e85d0a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:37.261220 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:37.261199    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:38.646275 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:38.646243    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:38.729505 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:38.729458    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:38.735496 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:38.735477    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:38.747192 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:38.747166    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:38.847790 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:38.847741    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:38.948404 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:38.948319    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:39.015432 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.015409    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:39.049129 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:39.049103    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-138.ec2.internal\" not found"
Jul 29 10:28:39.083295 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.083266    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 10:28:39.083674 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.083438    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 10:28:39.083674 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.083486    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 10:28:39.148753 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.148724    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:39.156372 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.156334    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 10:23:38 +0000 UTC" deadline="2028-03-16 09:08:30.655585474 +0000 UTC"
Jul 29 10:28:39.156372 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.156365    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14302h39m51.499223752s"
Jul 29 10:28:39.240534 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.240498    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:39.251520 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.251494    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:39.252649 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.252632    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal"
Jul 29 10:28:39.260535 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.260508    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:39.322023 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:39.321955    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda30ffe3ac8c128d369bee685e85d0a6.slice/crio-1e98e5d3f7d3a76bcd997e748d0c3034680ae61e272cd32fd182ea117bfe2005 WatchSource:0}: Error finding container 1e98e5d3f7d3a76bcd997e748d0c3034680ae61e272cd32fd182ea117bfe2005: Status 404 returned error can't find the container with id 1e98e5d3f7d3a76bcd997e748d0c3034680ae61e272cd32fd182ea117bfe2005
Jul 29 10:28:39.322466 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:39.322447    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e4af55a69b4ca5f48ea6f06887aafd.slice/crio-6309b2eaae6e30604965711b91c998a7065b24615f5dd31f6b4cbbade9271f74 WatchSource:0}: Error finding container 6309b2eaae6e30604965711b91c998a7065b24615f5dd31f6b4cbbade9271f74: Status 404 returned error can't find the container with id 6309b2eaae6e30604965711b91c998a7065b24615f5dd31f6b4cbbade9271f74
Jul 29 10:28:39.326871 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.326852    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:28:39.341221 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:39.341194    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:40.117938 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.117892    2572 apiserver.go:52] "Watching apiserver"
Jul 29 10:28:40.124524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.124309    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 10:28:40.124806 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.124777    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-p44gh","openshift-multus/network-metrics-daemon-l8kkx","openshift-network-operator/iptables-alerter-l5szg","kube-system/konnectivity-agent-jrqzr","kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal","openshift-dns/node-resolver-fns2s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal","openshift-network-diagnostics/network-check-target-wjnft","openshift-ovn-kubernetes/ovnkube-node-2hvgf","kube-system/global-pull-secret-syncer-fcpjf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz","openshift-cluster-node-tuning-operator/tuned-wq2vp","openshift-image-registry/node-ca-xgkvk","openshift-multus/multus-additional-cni-plugins-c4t8m"]
Jul 29 10:28:40.126657 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.126631    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.126773 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.126720    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:40.127985 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.127957    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.129789 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.129767    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f6gmd\""
Jul 29 10:28:40.129909 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.129832    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 10:28:40.130000 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.129765    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 10:28:40.130052 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.130023    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.131044 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.130834    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 10:28:40.131712 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.131690    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:40.132455 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.132082    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dd4qn\""
Jul 29 10:28:40.132455 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.132210    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:40.132455 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.132316    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 10:28:40.132646 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.132487    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:28:40.134274 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.134254    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.135904 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.135883    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 10:28:40.136235 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.136215    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zfqdm\""
Jul 29 10:28:40.136405 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.136390    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 10:28:40.137284 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.136990    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 10:28:40.137284 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.137234    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6wgb2\""
Jul 29 10:28:40.137425 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.137407    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 10:28:40.137569 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.137550    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.137624 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.137604    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 10:28:40.137846 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.137830    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 10:28:40.137904 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.137867    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 10:28:40.139139 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.138531    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.139425 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.139405    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 10:28:40.139885 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.139756    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 10:28:40.139885 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.139776    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x4d6r\""
Jul 29 10:28:40.140798 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.140482    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:40.140798 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.140784    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.141075 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.141055    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 10:28:40.141377 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.141359    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bzm6b\""
Jul 29 10:28:40.141471 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.141403    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 10:28:40.141543 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.141526    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 10:28:40.141623 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.141359    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 10:28:40.141623 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.141362    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 10:28:40.142049 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.141917    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 10:28:40.143021 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.142991    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 10:28:40.143293 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.143275    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzstd\""
Jul 29 10:28:40.145117 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.145094    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 10:28:40.145896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.145880    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 10:28:40.148996 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.147935    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:40.148996 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.148024    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:28:40.150408 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.150357    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.151227 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.151206    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:40.151825 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.151807    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:40.151916 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.151868    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:40.152326 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152303    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgjlw\" (UniqueName: \"kubernetes.io/projected/d1e36009-d378-43ca-8aef-e3245dab1ed4-kube-api-access-pgjlw\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.152410 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152362    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-slash\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152410 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-node-log\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152536 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3b896e1-b046-4eea-a864-6169f95a415f-ovnkube-script-lib\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152536 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152429    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb7wh\" (UniqueName: \"kubernetes.io/projected/a2f82d39-8f16-4606-a6d5-8f9ced950f85-kube-api-access-fb7wh\") pod \"node-ca-xgkvk\" (UID: \"a2f82d39-8f16-4606-a6d5-8f9ced950f85\") " pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.152536 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152445    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-run-ovn\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152536 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152459    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d900008f-ab70-4ed3-8521-a2098f40ae4c-agent-certs\") pod \"konnectivity-agent-jrqzr\" (UID: \"d900008f-ab70-4ed3-8521-a2098f40ae4c\") " pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:28:40.152536 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152473    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-system-cni-dir\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.152536 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152489    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c584cc-8c25-427e-9209-d70836610d73-cni-binary-copy\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.152536 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152510    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-systemd-units\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152532    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-etc-openvswitch\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152564    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-cni-bin\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152579    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3b896e1-b046-4eea-a864-6169f95a415f-ovn-node-metrics-cert\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152592    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8471251-f5f4-48f8-91d6-7864fbc9db94-host-slash\") pod \"iptables-alerter-l5szg\" (UID: \"e8471251-f5f4-48f8-91d6-7864fbc9db94\") " pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152607    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5w7r\" (UniqueName: \"kubernetes.io/projected/e8471251-f5f4-48f8-91d6-7864fbc9db94-kube-api-access-s5w7r\") pod \"iptables-alerter-l5szg\" (UID: \"e8471251-f5f4-48f8-91d6-7864fbc9db94\") " pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56c584cc-8c25-427e-9209-d70836610d73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-run-netns\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152687    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-kubelet-config\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152734    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152783    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d900008f-ab70-4ed3-8521-a2098f40ae4c-konnectivity-ca\") pod \"konnectivity-agent-jrqzr\" (UID: \"d900008f-ab70-4ed3-8521-a2098f40ae4c\") " pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152818    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtzj\" (UniqueName: \"kubernetes.io/projected/56c584cc-8c25-427e-9209-d70836610d73-kube-api-access-2xtzj\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152841    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-cni-netd\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.152873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152864    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-dbus\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152890    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-registration-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-device-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152948    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8799dbf0-e093-403b-997d-6bc3b2323a96-hosts-file\") pod \"node-resolver-fns2s\" (UID: \"8799dbf0-e093-403b-997d-6bc3b2323a96\") " pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlvsz\" (UniqueName: \"kubernetes.io/projected/8799dbf0-e093-403b-997d-6bc3b2323a96-kube-api-access-xlvsz\") pod \"node-resolver-fns2s\" (UID: \"8799dbf0-e093-403b-997d-6bc3b2323a96\") " pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.152987    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56c584cc-8c25-427e-9209-d70836610d73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153002    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-kubelet\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153015    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-run-systemd\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153030    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-var-lib-openvswitch\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153084    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3b896e1-b046-4eea-a864-6169f95a415f-ovnkube-config\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-socket-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153139    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2f82d39-8f16-4606-a6d5-8f9ced950f85-serviceca\") pod \"node-ca-xgkvk\" (UID: \"a2f82d39-8f16-4606-a6d5-8f9ced950f85\") " pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153164    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153192    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8471251-f5f4-48f8-91d6-7864fbc9db94-iptables-alerter-script\") pod \"iptables-alerter-l5szg\" (UID: \"e8471251-f5f4-48f8-91d6-7864fbc9db94\") " pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153217    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-os-release\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.153830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153246    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-run-openvswitch\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153271    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3b896e1-b046-4eea-a864-6169f95a415f-env-overrides\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153295    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153316    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2f82d39-8f16-4606-a6d5-8f9ced950f85-host\") pod \"node-ca-xgkvk\" (UID: \"a2f82d39-8f16-4606-a6d5-8f9ced950f85\") " pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153341    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-log-socket\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153350    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153373    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8799dbf0-e093-403b-997d-6bc3b2323a96-tmp-dir\") pod \"node-resolver-fns2s\" (UID: \"8799dbf0-e093-403b-997d-6bc3b2323a96\") " pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-cnibin\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153425    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbm7\" (UniqueName: \"kubernetes.io/projected/b3b896e1-b046-4eea-a864-6169f95a415f-kube-api-access-fwbm7\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153450    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-sys-fs\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.153670    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pvj7l\""
Jul 29 10:28:40.154628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.154441    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.156850 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.156451    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-52mkr\""
Jul 29 10:28:40.156850 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.156505    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:40.156850 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.156719    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:40.173543 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.173514    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-trhkn"
Jul 29 10:28:40.182801 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.182478    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-trhkn"
Jul 29 10:28:40.241464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.241434    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 10:28:40.253825 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.253794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-system-cni-dir\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.253825 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.253830    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-systemd-units\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.254109 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.253863    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-cni-bin\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.254109 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.253887    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.254109 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.253916    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-multus-conf-dir\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.254109 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.253913    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-system-cni-dir\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.254109 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.253986    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-systemd-units\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.254109 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254029    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.254109 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.253997    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-cni-bin\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.254109 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb7wh\" (UniqueName: \"kubernetes.io/projected/a2f82d39-8f16-4606-a6d5-8f9ced950f85-kube-api-access-fb7wh\") pod \"node-ca-xgkvk\" (UID: \"a2f82d39-8f16-4606-a6d5-8f9ced950f85\") " pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-modprobe-d\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8471251-f5f4-48f8-91d6-7864fbc9db94-host-slash\") pod \"iptables-alerter-l5szg\" (UID: \"e8471251-f5f4-48f8-91d6-7864fbc9db94\") " pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254191    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-registration-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254217    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-sysctl-d\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254240    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-var-lib-kubelet\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254272    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-system-cni-dir\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254296    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-os-release\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtzj\" (UniqueName: \"kubernetes.io/projected/56c584cc-8c25-427e-9209-d70836610d73-kube-api-access-2xtzj\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254347    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-cni-netd\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254371    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-dbus\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-device-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.254464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254419    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-run\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254474    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/779f2766-886b-4d01-9fdc-b54073e8604b-cni-binary-copy\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254511    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3b896e1-b046-4eea-a864-6169f95a415f-ovnkube-config\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254538    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-run-netns\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254547    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8471251-f5f4-48f8-91d6-7864fbc9db94-host-slash\") pod \"iptables-alerter-l5szg\" (UID: \"e8471251-f5f4-48f8-91d6-7864fbc9db94\") " pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqjk\" (UniqueName: \"kubernetes.io/projected/779f2766-886b-4d01-9fdc-b54073e8604b-kube-api-access-fhqjk\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254583    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-cni-netd\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254604    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254645    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3b896e1-b046-4eea-a864-6169f95a415f-env-overrides\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254667    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-registration-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-sysctl-conf\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-log-socket\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254716    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-device-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254763    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-multus-socket-dir-parent\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254785    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-var-lib-kubelet\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254808    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-cnibin\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbm7\" (UniqueName: \"kubernetes.io/projected/b3b896e1-b046-4eea-a864-6169f95a415f-kube-api-access-fwbm7\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254865    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254869    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-dbus\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254895    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-sys-fs\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254879    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-log-socket\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.254954    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-cnibin\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255263    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3b896e1-b046-4eea-a864-6169f95a415f-env-overrides\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255337    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3b896e1-b046-4eea-a864-6169f95a415f-ovnkube-config\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255537    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/779f2766-886b-4d01-9fdc-b54073e8604b-multus-daemon-config\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255557    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-sys-fs\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255611    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d900008f-ab70-4ed3-8521-a2098f40ae4c-agent-certs\") pod \"konnectivity-agent-jrqzr\" (UID: \"d900008f-ab70-4ed3-8521-a2098f40ae4c\") " pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255649    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c584cc-8c25-427e-9209-d70836610d73-cni-binary-copy\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255673    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-etc-openvswitch\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3b896e1-b046-4eea-a864-6169f95a415f-ovn-node-metrics-cert\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255757    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.255875    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-etc-openvswitch\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.255901 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.255902    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:40.256583 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.256361    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.256017    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret podName:b2f0ed09-a52b-4305-8fa6-435794cc0ff3 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:40.755973905 +0000 UTC m=+2.984220456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret") pod "global-pull-secret-syncer-fcpjf" (UID: "b2f0ed09-a52b-4305-8fa6-435794cc0ff3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.257577    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c584cc-8c25-427e-9209-d70836610d73-cni-binary-copy\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.257648    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-sysconfig\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.257697    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-kubernetes\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.257722    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-tuned\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.257772    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5w7r\" (UniqueName: \"kubernetes.io/projected/e8471251-f5f4-48f8-91d6-7864fbc9db94-kube-api-access-s5w7r\") pod \"iptables-alerter-l5szg\" (UID: \"e8471251-f5f4-48f8-91d6-7864fbc9db94\") " pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.257804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56c584cc-8c25-427e-9209-d70836610d73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.257832    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-run-netns\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.257999    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258014    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-run-netns\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258063    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-kubelet-config\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258305    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56c584cc-8c25-427e-9209-d70836610d73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8799dbf0-e093-403b-997d-6bc3b2323a96-hosts-file\") pod \"node-resolver-fns2s\" (UID: \"8799dbf0-e093-403b-997d-6bc3b2323a96\") " pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258372    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-kubelet-config\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258382    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlvsz\" (UniqueName: \"kubernetes.io/projected/8799dbf0-e093-403b-997d-6bc3b2323a96-kube-api-access-xlvsz\") pod \"node-resolver-fns2s\" (UID: \"8799dbf0-e093-403b-997d-6bc3b2323a96\") " pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.260476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258437    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8799dbf0-e093-403b-997d-6bc3b2323a96-hosts-file\") pod \"node-resolver-fns2s\" (UID: \"8799dbf0-e093-403b-997d-6bc3b2323a96\") " pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258477    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258526    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2f82d39-8f16-4606-a6d5-8f9ced950f85-host\") pod \"node-ca-xgkvk\" (UID: \"a2f82d39-8f16-4606-a6d5-8f9ced950f85\") " pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258564    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d900008f-ab70-4ed3-8521-a2098f40ae4c-konnectivity-ca\") pod \"konnectivity-agent-jrqzr\" (UID: \"d900008f-ab70-4ed3-8521-a2098f40ae4c\") " pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258600    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgmsz\" (UniqueName: \"kubernetes.io/projected/7be3cb65-5879-412b-b438-99e55616cf3e-kube-api-access-rgmsz\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-run-multus-certs\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258730    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56c584cc-8c25-427e-9209-d70836610d73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258759    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-kubelet\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.258977    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-kubelet\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259027    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2f82d39-8f16-4606-a6d5-8f9ced950f85-host\") pod \"node-ca-xgkvk\" (UID: \"a2f82d39-8f16-4606-a6d5-8f9ced950f85\") " pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259079    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-run-systemd\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259129    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-var-lib-openvswitch\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259193    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-socket-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2f82d39-8f16-4606-a6d5-8f9ced950f85-serviceca\") pod \"node-ca-xgkvk\" (UID: \"a2f82d39-8f16-4606-a6d5-8f9ced950f85\") " pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259278    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259304    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d900008f-ab70-4ed3-8521-a2098f40ae4c-konnectivity-ca\") pod \"konnectivity-agent-jrqzr\" (UID: \"d900008f-ab70-4ed3-8521-a2098f40ae4c\") " pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:28:40.261296 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8471251-f5f4-48f8-91d6-7864fbc9db94-iptables-alerter-script\") pod \"iptables-alerter-l5szg\" (UID: \"e8471251-f5f4-48f8-91d6-7864fbc9db94\") " pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259353    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-os-release\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259619    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259626    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-var-lib-openvswitch\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259677    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-run-openvswitch\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259717    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56c584cc-8c25-427e-9209-d70836610d73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259761    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-systemd\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259794    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-sys\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259856    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-host\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259896    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8799dbf0-e093-403b-997d-6bc3b2323a96-tmp-dir\") pod \"node-resolver-fns2s\" (UID: \"8799dbf0-e093-403b-997d-6bc3b2323a96\") " pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259941    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a9b3a64-a024-4aee-850c-d0975fc62eb5-tmp\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcf49\" (UniqueName: \"kubernetes.io/projected/8a9b3a64-a024-4aee-850c-d0975fc62eb5-kube-api-access-tcf49\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.259995    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260027    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgjlw\" (UniqueName: \"kubernetes.io/projected/d1e36009-d378-43ca-8aef-e3245dab1ed4-kube-api-access-pgjlw\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260175    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-multus-cni-dir\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260196    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8471251-f5f4-48f8-91d6-7864fbc9db94-iptables-alerter-script\") pod \"iptables-alerter-l5szg\" (UID: \"e8471251-f5f4-48f8-91d6-7864fbc9db94\") " pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.262043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260204    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-var-lib-cni-bin\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-hostroot\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260269    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-slash\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260301    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-node-log\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260328    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3b896e1-b046-4eea-a864-6169f95a415f-ovnkube-script-lib\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260355    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-lib-modules\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260375    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-cnibin\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260445    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-run-k8s-cni-cncf-io\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-var-lib-cni-multus\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260508    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-etc-kubernetes\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260533    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-run-ovn\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260635    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-run-ovn\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.260634    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2f82d39-8f16-4606-a6d5-8f9ced950f85-serviceca\") pod \"node-ca-xgkvk\" (UID: \"a2f82d39-8f16-4606-a6d5-8f9ced950f85\") " pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261015    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56c584cc-8c25-427e-9209-d70836610d73-os-release\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261128    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-socket-dir\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261175    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-run-systemd\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261217    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-run-openvswitch\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261229    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d900008f-ab70-4ed3-8521-a2098f40ae4c-agent-certs\") pod \"konnectivity-agent-jrqzr\" (UID: \"d900008f-ab70-4ed3-8521-a2098f40ae4c\") " pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:28:40.263096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261298    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1e36009-d378-43ca-8aef-e3245dab1ed4-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.264092 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261344    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-host-slash\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.264092 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261388    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3b896e1-b046-4eea-a864-6169f95a415f-node-log\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.264092 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261612    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8799dbf0-e093-403b-997d-6bc3b2323a96-tmp-dir\") pod \"node-resolver-fns2s\" (UID: \"8799dbf0-e093-403b-997d-6bc3b2323a96\") " pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.264092 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.261873    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3b896e1-b046-4eea-a864-6169f95a415f-ovnkube-script-lib\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.264286 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.264106    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtzj\" (UniqueName: \"kubernetes.io/projected/56c584cc-8c25-427e-9209-d70836610d73-kube-api-access-2xtzj\") pod \"multus-additional-cni-plugins-c4t8m\" (UID: \"56c584cc-8c25-427e-9209-d70836610d73\") " pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.264408 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.264384    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb7wh\" (UniqueName: \"kubernetes.io/projected/a2f82d39-8f16-4606-a6d5-8f9ced950f85-kube-api-access-fb7wh\") pod \"node-ca-xgkvk\" (UID: \"a2f82d39-8f16-4606-a6d5-8f9ced950f85\") " pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.266384 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.266358    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3b896e1-b046-4eea-a864-6169f95a415f-ovn-node-metrics-cert\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.266989 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.266965    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbm7\" (UniqueName: \"kubernetes.io/projected/b3b896e1-b046-4eea-a864-6169f95a415f-kube-api-access-fwbm7\") pod \"ovnkube-node-2hvgf\" (UID: \"b3b896e1-b046-4eea-a864-6169f95a415f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.267776 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.267740    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5w7r\" (UniqueName: \"kubernetes.io/projected/e8471251-f5f4-48f8-91d6-7864fbc9db94-kube-api-access-s5w7r\") pod \"iptables-alerter-l5szg\" (UID: \"e8471251-f5f4-48f8-91d6-7864fbc9db94\") " pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.269445 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.269411    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlvsz\" (UniqueName: \"kubernetes.io/projected/8799dbf0-e093-403b-997d-6bc3b2323a96-kube-api-access-xlvsz\") pod \"node-resolver-fns2s\" (UID: \"8799dbf0-e093-403b-997d-6bc3b2323a96\") " pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.277103 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.276945    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgjlw\" (UniqueName: \"kubernetes.io/projected/d1e36009-d378-43ca-8aef-e3245dab1ed4-kube-api-access-pgjlw\") pod \"aws-ebs-csi-driver-node-h4pwz\" (UID: \"d1e36009-d378-43ca-8aef-e3245dab1ed4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.309661 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.309601    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal" event={"ID":"da30ffe3ac8c128d369bee685e85d0a6","Type":"ContainerStarted","Data":"1e98e5d3f7d3a76bcd997e748d0c3034680ae61e272cd32fd182ea117bfe2005"}
Jul 29 10:28:40.310881 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.310838    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal" event={"ID":"a3e4af55a69b4ca5f48ea6f06887aafd","Type":"ContainerStarted","Data":"6309b2eaae6e30604965711b91c998a7065b24615f5dd31f6b4cbbade9271f74"}
Jul 29 10:28:40.361804 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-sysconfig\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.361804 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-kubernetes\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.361804 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-tuned\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgmsz\" (UniqueName: \"kubernetes.io/projected/7be3cb65-5879-412b-b438-99e55616cf3e-kube-api-access-rgmsz\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361878    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-run-multus-certs\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361908    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361935    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-kubernetes\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361950    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-systemd\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361936    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-sysconfig\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361976    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-sys\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.361978    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-run-multus-certs\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362027    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-systemd\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362027    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-host\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a9b3a64-a024-4aee-850c-d0975fc62eb5-tmp\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362077    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-host\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcf49\" (UniqueName: \"kubernetes.io/projected/8a9b3a64-a024-4aee-850c-d0975fc62eb5-kube-api-access-tcf49\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362125    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362173    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-multus-cni-dir\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-var-lib-cni-bin\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.362210 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362219    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-hostroot\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362244    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-lib-modules\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362268    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-cnibin\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-run-k8s-cni-cncf-io\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-var-lib-cni-multus\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362326    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-var-lib-cni-bin\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362339    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-etc-kubernetes\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-multus-conf-dir\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362393    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-modprobe-d\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.362409    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362420    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-sysctl-d\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362430    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-cnibin\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362434    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-lib-modules\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362444    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-var-lib-kubelet\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.362485    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs podName:7be3cb65-5879-412b-b438-99e55616cf3e nodeName:}" failed. No retries permitted until 2026-07-29 10:28:40.862445154 +0000 UTC m=+3.090691686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs") pod "network-metrics-daemon-l8kkx" (UID: "7be3cb65-5879-412b-b438-99e55616cf3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362492    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-hostroot\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362496    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-run-k8s-cni-cncf-io\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362503    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-multus-conf-dir\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362289    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-sys\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362542    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-multus-cni-dir\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362544    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-var-lib-kubelet\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362585    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-var-lib-cni-multus\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362592    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-system-cni-dir\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362616    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-modprobe-d\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362669    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-etc-kubernetes\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362698    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-os-release\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362739    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-run\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362759    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-system-cni-dir\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362763    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/779f2766-886b-4d01-9fdc-b54073e8604b-cni-binary-copy\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362798    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-os-release\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362808    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-run-netns\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362841    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqjk\" (UniqueName: \"kubernetes.io/projected/779f2766-886b-4d01-9fdc-b54073e8604b-kube-api-access-fhqjk\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-run-netns\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362834    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-run\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362868    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-sysctl-conf\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362892    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-multus-socket-dir-parent\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.363781 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362916    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-var-lib-kubelet\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.364476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.362959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/779f2766-886b-4d01-9fdc-b54073e8604b-multus-daemon-config\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.364476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.363060    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-sysctl-conf\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.364476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.363118    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-multus-socket-dir-parent\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.364476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.363126    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/779f2766-886b-4d01-9fdc-b54073e8604b-host-var-lib-kubelet\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.364476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.363222    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-sysctl-d\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.364476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.363399    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/779f2766-886b-4d01-9fdc-b54073e8604b-cni-binary-copy\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.364476 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.363535    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/779f2766-886b-4d01-9fdc-b54073e8604b-multus-daemon-config\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.364712 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.364495    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a9b3a64-a024-4aee-850c-d0975fc62eb5-tmp\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.364712 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.364632    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a9b3a64-a024-4aee-850c-d0975fc62eb5-etc-tuned\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.368866 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.368612    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:40.368866 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.368638    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:40.368866 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.368654    2572 projected.go:196] Error preparing data for projected volume kube-api-access-f969l for pod openshift-network-diagnostics/network-check-target-wjnft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:40.368866 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.368747    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l podName:084f30e4-3825-48a8-96ca-c5ebc0c10b5a nodeName:}" failed. No retries permitted until 2026-07-29 10:28:40.868727066 +0000 UTC m=+3.096973593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f969l" (UniqueName: "kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l") pod "network-check-target-wjnft" (UID: "084f30e4-3825-48a8-96ca-c5ebc0c10b5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:40.370447 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.370421    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqjk\" (UniqueName: \"kubernetes.io/projected/779f2766-886b-4d01-9fdc-b54073e8604b-kube-api-access-fhqjk\") pod \"multus-p44gh\" (UID: \"779f2766-886b-4d01-9fdc-b54073e8604b\") " pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.370564 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.370515    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgmsz\" (UniqueName: \"kubernetes.io/projected/7be3cb65-5879-412b-b438-99e55616cf3e-kube-api-access-rgmsz\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:40.371209 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.371192    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcf49\" (UniqueName: \"kubernetes.io/projected/8a9b3a64-a024-4aee-850c-d0975fc62eb5-kube-api-access-tcf49\") pod \"tuned-wq2vp\" (UID: \"8a9b3a64-a024-4aee-850c-d0975fc62eb5\") " pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.403783 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.403753    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:40.442690 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.442655    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz"
Jul 29 10:28:40.459316 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.459284    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l5szg"
Jul 29 10:28:40.467195 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.467157    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:28:40.475392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.475363    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c4t8m"
Jul 29 10:28:40.484226 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.484193    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fns2s"
Jul 29 10:28:40.492119 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.492093    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:28:40.499005 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.498965    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xgkvk"
Jul 29 10:28:40.506902 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.506866    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p44gh"
Jul 29 10:28:40.512643 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.512625    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wq2vp"
Jul 29 10:28:40.766032 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.765917    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:40.766210 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.766063    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:40.766210 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.766134    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret podName:b2f0ed09-a52b-4305-8fa6-435794cc0ff3 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:41.766118818 +0000 UTC m=+3.994365346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret") pod "global-pull-secret-syncer-fcpjf" (UID: "b2f0ed09-a52b-4305-8fa6-435794cc0ff3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:40.866717 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.866667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:40.866908 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.866826    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:40.866908 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.866899    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs podName:7be3cb65-5879-412b-b438-99e55616cf3e nodeName:}" failed. No retries permitted until 2026-07-29 10:28:41.866880681 +0000 UTC m=+4.095127211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs") pod "network-metrics-daemon-l8kkx" (UID: "7be3cb65-5879-412b-b438-99e55616cf3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:40.967221 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:40.967193    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:40.967407 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.967368    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:40.967407 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.967395    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:40.967407 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.967405    2572 projected.go:196] Error preparing data for projected volume kube-api-access-f969l for pod openshift-network-diagnostics/network-check-target-wjnft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:40.967622 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:40.967463    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l podName:084f30e4-3825-48a8-96ca-c5ebc0c10b5a nodeName:}" failed. No retries permitted until 2026-07-29 10:28:41.967448684 +0000 UTC m=+4.195695212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f969l" (UniqueName: "kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l") pod "network-check-target-wjnft" (UID: "084f30e4-3825-48a8-96ca-c5ebc0c10b5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:41.185325 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.185272    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:40 +0000 UTC" deadline="2028-02-03 14:33:50.336118829 +0000 UTC"
Jul 29 10:28:41.185325 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.185322    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13300h5m9.150805565s"
Jul 29 10:28:41.218178 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.218145    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:41.272040 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:41.272007    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e36009_d378_43ca_8aef_e3245dab1ed4.slice/crio-029e5e0efdd8fa387fe4956cd624b540befc79b687f2682eee0541ac103cfffe WatchSource:0}: Error finding container 029e5e0efdd8fa387fe4956cd624b540befc79b687f2682eee0541ac103cfffe: Status 404 returned error can't find the container with id 029e5e0efdd8fa387fe4956cd624b540befc79b687f2682eee0541ac103cfffe
Jul 29 10:28:41.273567 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:41.273541    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b896e1_b046_4eea_a864_6169f95a415f.slice/crio-c17399cb2206a5cf81b310fc57769601b0198f69870ee618c342dd633ce6ea36 WatchSource:0}: Error finding container c17399cb2206a5cf81b310fc57769601b0198f69870ee618c342dd633ce6ea36: Status 404 returned error can't find the container with id c17399cb2206a5cf81b310fc57769601b0198f69870ee618c342dd633ce6ea36
Jul 29 10:28:41.280560 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:41.280530    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c584cc_8c25_427e_9209_d70836610d73.slice/crio-cd36515fe64d58d83e16ab04a89e524d7ab475c969d3a7be822d8d988f830dcc WatchSource:0}: Error finding container cd36515fe64d58d83e16ab04a89e524d7ab475c969d3a7be822d8d988f830dcc: Status 404 returned error can't find the container with id cd36515fe64d58d83e16ab04a89e524d7ab475c969d3a7be822d8d988f830dcc
Jul 29 10:28:41.281340 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:41.281318    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd900008f_ab70_4ed3_8521_a2098f40ae4c.slice/crio-9f88b41ba0831dbf58e5be6a6ca72deaba84f2b31b0b5e08dd21e4b043188d94 WatchSource:0}: Error finding container 9f88b41ba0831dbf58e5be6a6ca72deaba84f2b31b0b5e08dd21e4b043188d94: Status 404 returned error can't find the container with id 9f88b41ba0831dbf58e5be6a6ca72deaba84f2b31b0b5e08dd21e4b043188d94
Jul 29 10:28:41.283091 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:41.282976    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779f2766_886b_4d01_9fdc_b54073e8604b.slice/crio-0f492a9df3794985a0811aafe538f3ba83b8c0d707d080494e6eb4d6e4cb022b WatchSource:0}: Error finding container 0f492a9df3794985a0811aafe538f3ba83b8c0d707d080494e6eb4d6e4cb022b: Status 404 returned error can't find the container with id 0f492a9df3794985a0811aafe538f3ba83b8c0d707d080494e6eb4d6e4cb022b
Jul 29 10:28:41.283977 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:41.283954    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9b3a64_a024_4aee_850c_d0975fc62eb5.slice/crio-9f357390dc51e2b013ee433fe15de05da9578e53e37735cb40862143d55ddbf6 WatchSource:0}: Error finding container 9f357390dc51e2b013ee433fe15de05da9578e53e37735cb40862143d55ddbf6: Status 404 returned error can't find the container with id 9f357390dc51e2b013ee433fe15de05da9578e53e37735cb40862143d55ddbf6
Jul 29 10:28:41.294857 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:41.294832    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f82d39_8f16_4606_a6d5_8f9ced950f85.slice/crio-abd225b4cc6741713023d9ab634636748a58fce60ea6285981ac1238697e3c2b WatchSource:0}: Error finding container abd225b4cc6741713023d9ab634636748a58fce60ea6285981ac1238697e3c2b: Status 404 returned error can't find the container with id abd225b4cc6741713023d9ab634636748a58fce60ea6285981ac1238697e3c2b
Jul 29 10:28:41.296406 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:28:41.296349    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8471251_f5f4_48f8_91d6_7864fbc9db94.slice/crio-be77e3c6a135fb8997142e31f2494cf7bd0cc9ebaae1e9a14809571e8a6edb18 WatchSource:0}: Error finding container be77e3c6a135fb8997142e31f2494cf7bd0cc9ebaae1e9a14809571e8a6edb18: Status 404 returned error can't find the container with id be77e3c6a135fb8997142e31f2494cf7bd0cc9ebaae1e9a14809571e8a6edb18
Jul 29 10:28:41.313288 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.313258    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jrqzr" event={"ID":"d900008f-ab70-4ed3-8521-a2098f40ae4c","Type":"ContainerStarted","Data":"9f88b41ba0831dbf58e5be6a6ca72deaba84f2b31b0b5e08dd21e4b043188d94"}
Jul 29 10:28:41.314437 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.314405    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" event={"ID":"b3b896e1-b046-4eea-a864-6169f95a415f","Type":"ContainerStarted","Data":"c17399cb2206a5cf81b310fc57769601b0198f69870ee618c342dd633ce6ea36"}
Jul 29 10:28:41.319463 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.319434    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz" event={"ID":"d1e36009-d378-43ca-8aef-e3245dab1ed4","Type":"ContainerStarted","Data":"029e5e0efdd8fa387fe4956cd624b540befc79b687f2682eee0541ac103cfffe"}
Jul 29 10:28:41.320520 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.320489    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xgkvk" event={"ID":"a2f82d39-8f16-4606-a6d5-8f9ced950f85","Type":"ContainerStarted","Data":"abd225b4cc6741713023d9ab634636748a58fce60ea6285981ac1238697e3c2b"}
Jul 29 10:28:41.321876 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.321839    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c4t8m" event={"ID":"56c584cc-8c25-427e-9209-d70836610d73","Type":"ContainerStarted","Data":"cd36515fe64d58d83e16ab04a89e524d7ab475c969d3a7be822d8d988f830dcc"}
Jul 29 10:28:41.322972 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.322947    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fns2s" event={"ID":"8799dbf0-e093-403b-997d-6bc3b2323a96","Type":"ContainerStarted","Data":"85ea574892250c005aab50033ab9e7eaa144e5ac0068a2c62b7f2ef8267e030c"}
Jul 29 10:28:41.326453 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.326427    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l5szg" event={"ID":"e8471251-f5f4-48f8-91d6-7864fbc9db94","Type":"ContainerStarted","Data":"be77e3c6a135fb8997142e31f2494cf7bd0cc9ebaae1e9a14809571e8a6edb18"}
Jul 29 10:28:41.332114 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.332082    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wq2vp" event={"ID":"8a9b3a64-a024-4aee-850c-d0975fc62eb5","Type":"ContainerStarted","Data":"9f357390dc51e2b013ee433fe15de05da9578e53e37735cb40862143d55ddbf6"}
Jul 29 10:28:41.333816 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.333786    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p44gh" event={"ID":"779f2766-886b-4d01-9fdc-b54073e8604b","Type":"ContainerStarted","Data":"0f492a9df3794985a0811aafe538f3ba83b8c0d707d080494e6eb4d6e4cb022b"}
Jul 29 10:28:41.777152 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.776816    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:41.777363 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:41.777071    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:41.777363 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:41.777262    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret podName:b2f0ed09-a52b-4305-8fa6-435794cc0ff3 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:43.77723987 +0000 UTC m=+6.005486402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret") pod "global-pull-secret-syncer-fcpjf" (UID: "b2f0ed09-a52b-4305-8fa6-435794cc0ff3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:41.878200 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.878157    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:41.878397 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:41.878302    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:41.878397 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:41.878370    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs podName:7be3cb65-5879-412b-b438-99e55616cf3e nodeName:}" failed. No retries permitted until 2026-07-29 10:28:43.878350323 +0000 UTC m=+6.106596851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs") pod "network-metrics-daemon-l8kkx" (UID: "7be3cb65-5879-412b-b438-99e55616cf3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:41.978830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:41.978788    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:41.979031 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:41.979011    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:41.979120 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:41.979038    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:41.979120 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:41.979052    2572 projected.go:196] Error preparing data for projected volume kube-api-access-f969l for pod openshift-network-diagnostics/network-check-target-wjnft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:41.979120 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:41.979111    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l podName:084f30e4-3825-48a8-96ca-c5ebc0c10b5a nodeName:}" failed. No retries permitted until 2026-07-29 10:28:43.979092594 +0000 UTC m=+6.207339123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f969l" (UniqueName: "kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l") pod "network-check-target-wjnft" (UID: "084f30e4-3825-48a8-96ca-c5ebc0c10b5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:42.186552 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:42.186475    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:40 +0000 UTC" deadline="2028-03-29 10:04:46.425737127 +0000 UTC"
Jul 29 10:28:42.186552 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:42.186541    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14615h36m4.239199576s"
Jul 29 10:28:42.306095 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:42.306061    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:42.306288 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:42.306213    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:28:42.306627 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:42.306609    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:42.306719 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:42.306697    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:42.306782 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:42.306772    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:42.306859 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:42.306841    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:42.382967 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:42.382840    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal" event={"ID":"a3e4af55a69b4ca5f48ea6f06887aafd","Type":"ContainerStarted","Data":"05cada7c45ea502a2dd8ed5d5e6ea478c518b03e86f60b2c5c0fd04db8198995"}
Jul 29 10:28:42.404164 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:42.404051    2572 generic.go:356] "Generic (PLEG): container finished" podID="da30ffe3ac8c128d369bee685e85d0a6" containerID="ce7418d786487240175f915016510f1f59b01e850a9f78371c9985f2e260eefe" exitCode=0
Jul 29 10:28:42.404164 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:42.404109    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal" event={"ID":"da30ffe3ac8c128d369bee685e85d0a6","Type":"ContainerDied","Data":"ce7418d786487240175f915016510f1f59b01e850a9f78371c9985f2e260eefe"}
Jul 29 10:28:42.421517 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:42.421462    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-138.ec2.internal" podStartSLOduration=3.42144068 podStartE2EDuration="3.42144068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:28:42.401594265 +0000 UTC m=+4.629840818" watchObservedRunningTime="2026-07-29 10:28:42.42144068 +0000 UTC m=+4.649687231"
Jul 29 10:28:43.444670 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:43.444395    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal" event={"ID":"da30ffe3ac8c128d369bee685e85d0a6","Type":"ContainerStarted","Data":"c54e12d349818149ec7add091546ecedc59eb01671036ac75d326e9c58683703"}
Jul 29 10:28:43.797780 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:43.797070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:43.797780 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:43.797283    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:43.797780 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:43.797350    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret podName:b2f0ed09-a52b-4305-8fa6-435794cc0ff3 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:47.797330603 +0000 UTC m=+10.025577133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret") pod "global-pull-secret-syncer-fcpjf" (UID: "b2f0ed09-a52b-4305-8fa6-435794cc0ff3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:43.898242 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:43.898201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:43.898422 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:43.898399    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:43.898477 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:43.898467    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs podName:7be3cb65-5879-412b-b438-99e55616cf3e nodeName:}" failed. No retries permitted until 2026-07-29 10:28:47.898446415 +0000 UTC m=+10.126692947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs") pod "network-metrics-daemon-l8kkx" (UID: "7be3cb65-5879-412b-b438-99e55616cf3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:43.998836 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:43.998801    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:43.999056 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:43.998997    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:43.999056 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:43.999018    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:43.999056 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:43.999032    2572 projected.go:196] Error preparing data for projected volume kube-api-access-f969l for pod openshift-network-diagnostics/network-check-target-wjnft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:43.999225 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:43.999091    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l podName:084f30e4-3825-48a8-96ca-c5ebc0c10b5a nodeName:}" failed. No retries permitted until 2026-07-29 10:28:47.999071966 +0000 UTC m=+10.227318498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f969l" (UniqueName: "kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l") pod "network-check-target-wjnft" (UID: "084f30e4-3825-48a8-96ca-c5ebc0c10b5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:44.305349 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:44.304779    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:44.305349 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:44.304943    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:44.305349 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:44.305064    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:44.305349 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:44.305174    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:44.305349 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:44.305251    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:44.305349 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:44.305331    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:28:46.306066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:46.306030    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:46.306552 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:46.306178    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:28:46.306552 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:46.306524    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:46.306681 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:46.306626    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:46.306681 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:46.306670    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:46.306780 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:46.306736    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:47.832019 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:47.831971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:47.832499 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:47.832129    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:47.832499 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:47.832203    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret podName:b2f0ed09-a52b-4305-8fa6-435794cc0ff3 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:55.83218353 +0000 UTC m=+18.060430076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret") pod "global-pull-secret-syncer-fcpjf" (UID: "b2f0ed09-a52b-4305-8fa6-435794cc0ff3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:47.932858 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:47.932819    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:47.933062 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:47.932983    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:47.933062 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:47.933046    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs podName:7be3cb65-5879-412b-b438-99e55616cf3e nodeName:}" failed. No retries permitted until 2026-07-29 10:28:55.933027627 +0000 UTC m=+18.161274159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs") pod "network-metrics-daemon-l8kkx" (UID: "7be3cb65-5879-412b-b438-99e55616cf3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:48.033294 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:48.033256    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:48.033540 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:48.033439    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:48.033540 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:48.033460    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:48.033540 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:48.033473    2572 projected.go:196] Error preparing data for projected volume kube-api-access-f969l for pod openshift-network-diagnostics/network-check-target-wjnft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:48.033540 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:48.033528    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l podName:084f30e4-3825-48a8-96ca-c5ebc0c10b5a nodeName:}" failed. No retries permitted until 2026-07-29 10:28:56.033509579 +0000 UTC m=+18.261756122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f969l" (UniqueName: "kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l") pod "network-check-target-wjnft" (UID: "084f30e4-3825-48a8-96ca-c5ebc0c10b5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:48.306235 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:48.305494    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:48.306235 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:48.305616    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:48.306235 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:48.306020    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:48.306235 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:48.306193    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:28:48.306235 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:48.306211    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:48.306617 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:48.306293    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:50.304951 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:50.304803    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:50.304951 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:50.304844    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:50.304951 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:50.304803    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:50.305542 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:50.304981    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:28:50.306013 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:50.305960    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:50.306156 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:50.306034    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:52.304709 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:52.304667    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:52.304709 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:52.304705    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:52.305232 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:52.304667    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:52.305232 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:52.304814    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:52.305232 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:52.304902    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:52.305232 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:52.305019    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:28:54.304971 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:54.304898    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:54.305443 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:54.304903    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:54.305443 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:54.305056    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:54.305443 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:54.305053    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:54.305443 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:54.305189    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:28:54.305443 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:54.305271    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:55.895807 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:55.895771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:55.896307 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:55.895932    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:55.896307 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:55.895997    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret podName:b2f0ed09-a52b-4305-8fa6-435794cc0ff3 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:11.895980143 +0000 UTC m=+34.124226679 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret") pod "global-pull-secret-syncer-fcpjf" (UID: "b2f0ed09-a52b-4305-8fa6-435794cc0ff3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:55.996489 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:55.996444    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:55.996672 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:55.996637    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:55.996734 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:55.996721    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs podName:7be3cb65-5879-412b-b438-99e55616cf3e nodeName:}" failed. No retries permitted until 2026-07-29 10:29:11.996698898 +0000 UTC m=+34.224945440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs") pod "network-metrics-daemon-l8kkx" (UID: "7be3cb65-5879-412b-b438-99e55616cf3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:56.096999 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:56.096958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:56.097219 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:56.097148    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:56.097219 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:56.097170    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:56.097219 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:56.097184    2572 projected.go:196] Error preparing data for projected volume kube-api-access-f969l for pod openshift-network-diagnostics/network-check-target-wjnft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:56.097387 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:56.097250    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l podName:084f30e4-3825-48a8-96ca-c5ebc0c10b5a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:12.097229839 +0000 UTC m=+34.325476367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f969l" (UniqueName: "kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l") pod "network-check-target-wjnft" (UID: "084f30e4-3825-48a8-96ca-c5ebc0c10b5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:56.305205 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:56.305114    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:56.305418 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:56.305114    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:56.305418 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:56.305248    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:56.305418 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:56.305339    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:56.305418 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:56.305114    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:56.305602 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:56.305456    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:28:58.306392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:58.306358    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:28:58.306859 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:58.306475    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:28:58.306859 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:58.306495    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:28:58.306859 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:58.306549    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:28:58.306859 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:28:58.306602    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:28:58.306859 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:28:58.306681    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:29:00.305172 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.304864    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:00.305894 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.304863    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:00.305894 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:00.305281    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:29:00.305894 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.304863    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:00.305894 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:00.305332    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:29:00.305894 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:00.305390    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:29:00.482271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.482235    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wq2vp" event={"ID":"8a9b3a64-a024-4aee-850c-d0975fc62eb5","Type":"ContainerStarted","Data":"e3fd7c251981a26571ae53f091dd797a4153c58a861aae9dc7915a844b1b5344"}
Jul 29 10:29:00.483839 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.483804    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p44gh" event={"ID":"779f2766-886b-4d01-9fdc-b54073e8604b","Type":"ContainerStarted","Data":"148d08252c2c166c8715946a7edf47a3a65cc47a7b9070874c7fbca3eeadcf31"}
Jul 29 10:29:00.485356 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.485322    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jrqzr" event={"ID":"d900008f-ab70-4ed3-8521-a2098f40ae4c","Type":"ContainerStarted","Data":"860b750b7b6642c19416858b03b96504b5a548a75b67c7d86509a7b23a76ddef"}
Jul 29 10:29:00.488593 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.488567    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" event={"ID":"b3b896e1-b046-4eea-a864-6169f95a415f","Type":"ContainerStarted","Data":"c082a55dbc521a93a036ad074b692c4be2a735aedbc1b9c9cc8c9baac4232a88"}
Jul 29 10:29:00.488744 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.488639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" event={"ID":"b3b896e1-b046-4eea-a864-6169f95a415f","Type":"ContainerStarted","Data":"210a82a53a19fddd9fd450154b8ae76e171e6363d467226151499be399967214"}
Jul 29 10:29:00.488744 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.488692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" event={"ID":"b3b896e1-b046-4eea-a864-6169f95a415f","Type":"ContainerStarted","Data":"d0c65b6351126b7c374829df07ad209db5d87d6bbfc02e916f9441f5ff6f71a8"}
Jul 29 10:29:00.488744 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.488740    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" event={"ID":"b3b896e1-b046-4eea-a864-6169f95a415f","Type":"ContainerStarted","Data":"afee3befbf0bb40baa87fd4efa459cfc8a7110f0f978dfdb3684b752d142f8e0"}
Jul 29 10:29:00.488886 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.488788    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" event={"ID":"b3b896e1-b046-4eea-a864-6169f95a415f","Type":"ContainerStarted","Data":"1c799ca772e9de57264f120459e485a0306873cefcd7e38031593d695d381393"}
Jul 29 10:29:00.488886 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.488836    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" event={"ID":"b3b896e1-b046-4eea-a864-6169f95a415f","Type":"ContainerStarted","Data":"80e465f75de93bc590ee1af83064bff1fefb20a44ca2a1a5d6cc8b7776052194"}
Jul 29 10:29:00.490096 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.490069    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz" event={"ID":"d1e36009-d378-43ca-8aef-e3245dab1ed4","Type":"ContainerStarted","Data":"eca4945a1c5f48e8be7c387229f124639e1a7a25af74b25220eab7a60546812d"}
Jul 29 10:29:00.491645 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.491621    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xgkvk" event={"ID":"a2f82d39-8f16-4606-a6d5-8f9ced950f85","Type":"ContainerStarted","Data":"7c46d62deed7a254677e92177f18b672626b1e3dee5d7f5d6142a9d695ad47a6"}
Jul 29 10:29:00.493390 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.493365    2572 generic.go:356] "Generic (PLEG): container finished" podID="56c584cc-8c25-427e-9209-d70836610d73" containerID="053951a291aa7344679221bea95d36dbbdca6ce45fcf127be681383f99bafb8d" exitCode=0
Jul 29 10:29:00.493505 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.493451    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c4t8m" event={"ID":"56c584cc-8c25-427e-9209-d70836610d73","Type":"ContainerDied","Data":"053951a291aa7344679221bea95d36dbbdca6ce45fcf127be681383f99bafb8d"}
Jul 29 10:29:00.494987 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.494937    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fns2s" event={"ID":"8799dbf0-e093-403b-997d-6bc3b2323a96","Type":"ContainerStarted","Data":"bb7c7cc1d70d5a283b0b235d018eeb0e9b9e9bd9aec682c90ae47218f2b0d6fe"}
Jul 29 10:29:00.497797 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.497749    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-138.ec2.internal" podStartSLOduration=21.49770903 podStartE2EDuration="21.49770903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:28:43.463529215 +0000 UTC m=+5.691775766" watchObservedRunningTime="2026-07-29 10:29:00.49770903 +0000 UTC m=+22.725955575"
Jul 29 10:29:00.498266 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.498225    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wq2vp" podStartSLOduration=4.000939042 podStartE2EDuration="22.498214729s" totalImagesPullingTime="18.497275687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:00.497815104 +0000 UTC m=+22.726061645" watchObservedRunningTime="2026-07-29 10:29:00.498214729 +0000 UTC m=+22.726461282"
Jul 29 10:29:00.515343 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.515297    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jrqzr" podStartSLOduration=11.831792607 podStartE2EDuration="22.515282993s" totalImagesPullingTime="10.683490386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:00.514661175 +0000 UTC m=+22.742907724" watchObservedRunningTime="2026-07-29 10:29:00.515282993 +0000 UTC m=+22.743529543"
Jul 29 10:29:00.551164 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.551115    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fns2s" podStartSLOduration=4.041808741 podStartE2EDuration="22.551099077s" totalImagesPullingTime="18.509290336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:00.550801958 +0000 UTC m=+22.779048587" watchObservedRunningTime="2026-07-29 10:29:00.551099077 +0000 UTC m=+22.779345628"
Jul 29 10:29:00.566409 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.566270    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xgkvk" podStartSLOduration=4.074653195 podStartE2EDuration="22.566241153s" totalImagesPullingTime="18.491587958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:00.565264751 +0000 UTC m=+22.793511302" watchObservedRunningTime="2026-07-29 10:29:00.566241153 +0000 UTC m=+22.794487700"
Jul 29 10:29:00.583076 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:00.583005    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p44gh" podStartSLOduration=4.032976161 podStartE2EDuration="22.582980142s" totalImagesPullingTime="18.550003981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:00.58275893 +0000 UTC m=+22.811005490" watchObservedRunningTime="2026-07-29 10:29:00.582980142 +0000 UTC m=+22.811226693"
Jul 29 10:29:01.353904 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:01.353875    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 10:29:01.499518 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:01.499431    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l5szg" event={"ID":"e8471251-f5f4-48f8-91d6-7864fbc9db94","Type":"ContainerStarted","Data":"d2bd051e98586c6034486b10bc618d9f567038ac5c301ee6655015c138560344"}
Jul 29 10:29:01.501701 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:01.501657    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz" event={"ID":"d1e36009-d378-43ca-8aef-e3245dab1ed4","Type":"ContainerStarted","Data":"65226fb46c7612906e358c0b0fabb00a15559a413ebc9fff2e1d511e72bfe368"}
Jul 29 10:29:01.516199 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:01.516153    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l5szg" podStartSLOduration=5.026328996 podStartE2EDuration="23.516132555s" totalImagesPullingTime="18.489803559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:01.516056125 +0000 UTC m=+23.744302687" watchObservedRunningTime="2026-07-29 10:29:01.516132555 +0000 UTC m=+23.744379098"
Jul 29 10:29:01.956623 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:01.956584    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:29:01.957270 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:01.957218    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:29:02.222603 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:02.222435    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T10:29:01.353897572Z","UUID":"444618bb-a46a-4674-8f00-99e54e161e36","Handler":null,"Name":"","Endpoint":""}
Jul 29 10:29:02.224491 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:02.224463    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 10:29:02.224658 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:02.224510    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 10:29:02.304547 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:02.304510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:02.304750 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:02.304655    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:29:02.304750 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:02.304510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:02.304857 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:02.304758    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:29:02.304857 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:02.304517    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:02.304971 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:02.304867    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:29:02.504515 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:02.504409    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:29:02.504959 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:02.504807    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jrqzr"
Jul 29 10:29:03.511050 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:03.510805    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" event={"ID":"b3b896e1-b046-4eea-a864-6169f95a415f","Type":"ContainerStarted","Data":"db334b6aac3c606708dde0d5f6c9aa63d89b37cc72e9cf7858f798bbc0438a7b"}
Jul 29 10:29:03.513084 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:03.513051    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz" event={"ID":"d1e36009-d378-43ca-8aef-e3245dab1ed4","Type":"ContainerStarted","Data":"98d6b01c95211c6a5309f47fe04edc7da67780c04dcf9255a22561041b7decb2"}
Jul 29 10:29:04.305272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:04.305228    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:04.305452 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:04.305284    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:04.305452 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:04.305228    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:04.305452 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:04.305403    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:29:04.305627 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:04.305512    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:29:04.305690 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:04.305657    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:29:05.521887 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:05.521696    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" event={"ID":"b3b896e1-b046-4eea-a864-6169f95a415f","Type":"ContainerStarted","Data":"49c6c1d97bf3c7536f5582b105df199c8b564e4018ec36d5c94b2899868f6315"}
Jul 29 10:29:05.522707 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:05.521969    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:29:05.523699 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:05.523670    2572 generic.go:356] "Generic (PLEG): container finished" podID="56c584cc-8c25-427e-9209-d70836610d73" containerID="16c611ed631cb1d7d45cd190fc548546947496ce2c4b0576489828b9a8d058d5" exitCode=0
Jul 29 10:29:05.523821 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:05.523719    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c4t8m" event={"ID":"56c584cc-8c25-427e-9209-d70836610d73","Type":"ContainerDied","Data":"16c611ed631cb1d7d45cd190fc548546947496ce2c4b0576489828b9a8d058d5"}
Jul 29 10:29:05.538555 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:05.538528    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:29:05.549667 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:05.547382    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4pwz" podStartSLOduration=6.489731129 podStartE2EDuration="27.547361156s" totalImagesPullingTime="21.057630027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:03.530444315 +0000 UTC m=+25.758690864" watchObservedRunningTime="2026-07-29 10:29:05.547361156 +0000 UTC m=+27.775607707"
Jul 29 10:29:05.549667 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:05.548389    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf" podStartSLOduration=8.952337787 podStartE2EDuration="27.548374138s" totalImagesPullingTime="18.596036351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:05.547270752 +0000 UTC m=+27.775517305" watchObservedRunningTime="2026-07-29 10:29:05.548374138 +0000 UTC m=+27.776620692"
Jul 29 10:29:06.306111 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:06.306073    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:06.306328 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:06.306204    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:29:06.307017 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:06.306673    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:06.307017 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:06.306781    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:29:06.307017 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:06.306856    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:06.307017 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:06.306962    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:29:06.525827 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:06.525793    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 10:29:06.526310 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:06.526290    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:29:06.542212 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:06.542186    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:29:07.281207 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:07.281172    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wjnft"]
Jul 29 10:29:07.281386 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:07.281301    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:07.281444 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:07.281412    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:29:07.284602 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:07.284567    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fcpjf"]
Jul 29 10:29:07.284812 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:07.284694    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:07.284877 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:07.284808    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:29:07.285268 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:07.285244    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l8kkx"]
Jul 29 10:29:07.285386 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:07.285369    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:07.285530 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:07.285483    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:29:07.530786 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:07.530748    2572 generic.go:356] "Generic (PLEG): container finished" podID="56c584cc-8c25-427e-9209-d70836610d73" containerID="8422839a4ef1856ac0bc8c8e9b012bffea250fc7fc3b77c8bca2c96e38047e65" exitCode=0
Jul 29 10:29:07.531301 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:07.530792    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c4t8m" event={"ID":"56c584cc-8c25-427e-9209-d70836610d73","Type":"ContainerDied","Data":"8422839a4ef1856ac0bc8c8e9b012bffea250fc7fc3b77c8bca2c96e38047e65"}
Jul 29 10:29:07.531301 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:07.530891    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 10:29:08.380186 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:08.380131    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:29:09.304833 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:09.304625    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:09.305406 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:09.304643    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:09.305406 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:09.304936    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:29:09.305406 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:09.304645    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:09.305406 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:09.305073    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:29:09.305406 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:09.305132    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:29:09.538877 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:09.538778    2572 generic.go:356] "Generic (PLEG): container finished" podID="56c584cc-8c25-427e-9209-d70836610d73" containerID="4817c8734d7212f68f2e9fc80d29064940ead4b91891702e891984abd16836c2" exitCode=0
Jul 29 10:29:09.538877 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:09.538848    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c4t8m" event={"ID":"56c584cc-8c25-427e-9209-d70836610d73","Type":"ContainerDied","Data":"4817c8734d7212f68f2e9fc80d29064940ead4b91891702e891984abd16836c2"}
Jul 29 10:29:11.304368 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:11.304334    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:11.304945 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:11.304334    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:11.304945 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:11.304481    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:29:11.304945 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:11.304334    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:11.304945 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:11.304518    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:29:11.304945 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:11.304583    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:29:11.918050 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:11.918006    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:11.918253 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:11.918149    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:11.918253 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:11.918230    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret podName:b2f0ed09-a52b-4305-8fa6-435794cc0ff3 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:43.918210701 +0000 UTC m=+66.146457247 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret") pod "global-pull-secret-syncer-fcpjf" (UID: "b2f0ed09-a52b-4305-8fa6-435794cc0ff3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:12.018677 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:12.018635    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:12.018877 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:12.018795    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:12.018877 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:12.018875    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs podName:7be3cb65-5879-412b-b438-99e55616cf3e nodeName:}" failed. No retries permitted until 2026-07-29 10:29:44.018857692 +0000 UTC m=+66.247104220 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs") pod "network-metrics-daemon-l8kkx" (UID: "7be3cb65-5879-412b-b438-99e55616cf3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:12.119676 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:12.119638    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:12.119875 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:12.119826    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:12.119875 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:12.119842    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:12.119875 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:12.119852    2572 projected.go:196] Error preparing data for projected volume kube-api-access-f969l for pod openshift-network-diagnostics/network-check-target-wjnft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:12.120073 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:12.119908    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l podName:084f30e4-3825-48a8-96ca-c5ebc0c10b5a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:44.11988891 +0000 UTC m=+66.348135457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f969l" (UniqueName: "kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l") pod "network-check-target-wjnft" (UID: "084f30e4-3825-48a8-96ca-c5ebc0c10b5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:13.302537 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.302509    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fns2s_8799dbf0-e093-403b-997d-6bc3b2323a96/dns-node-resolver/0.log"
Jul 29 10:29:13.304740 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.304706    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:13.304886 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.304703    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:13.304886 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.304821    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:13.304886 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:13.304814    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjnft" podUID="084f30e4-3825-48a8-96ca-c5ebc0c10b5a"
Jul 29 10:29:13.305050 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:13.304949    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l8kkx" podUID="7be3cb65-5879-412b-b438-99e55616cf3e"
Jul 29 10:29:13.305101 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:13.305043    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fcpjf" podUID="b2f0ed09-a52b-4305-8fa6-435794cc0ff3"
Jul 29 10:29:13.563553 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.563460    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-138.ec2.internal" event="NodeReady"
Jul 29 10:29:13.563719 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.563629    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 10:29:13.609095 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.609055    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zmftl"]
Jul 29 10:29:13.613660 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.613633    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rwhkx"]
Jul 29 10:29:13.613797 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.613776    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.615734 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.615708    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rxxsv\""
Jul 29 10:29:13.615851 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.615732    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 10:29:13.615851 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.615717    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 10:29:13.617036 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.617016    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.618885 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.618864    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 10:29:13.618991 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.618962    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 10:29:13.619312 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.619290    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 10:29:13.619399 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.619292    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5579t\""
Jul 29 10:29:13.619399 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.619351    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 10:29:13.622529 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.622295    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmftl"]
Jul 29 10:29:13.624784 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.624764    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rwhkx"]
Jul 29 10:29:13.712014 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.711980    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g7l9f"]
Jul 29 10:29:13.715746 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.715718    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g7l9f"
Jul 29 10:29:13.717831 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.717809    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 10:29:13.718015 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.717813    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zl4tt\""
Jul 29 10:29:13.718431 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.718411    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 10:29:13.718522 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.718420    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 10:29:13.724762 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.724735    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g7l9f"]
Jul 29 10:29:13.735584 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.735551    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.735765 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.735593    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrgw\" (UniqueName: \"kubernetes.io/projected/090b067b-9649-4758-934b-459ca2bb383d-kube-api-access-8vrgw\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.735765 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.735629    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrv6\" (UniqueName: \"kubernetes.io/projected/1124f153-b0c0-4132-88a3-c60068f9e2c3-kube-api-access-4zrv6\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.735765 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.735686    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/090b067b-9649-4758-934b-459ca2bb383d-data-volume\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.735765 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.735712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.735765 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.735729    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1124f153-b0c0-4132-88a3-c60068f9e2c3-tmp-dir\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.735984 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.735806    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/090b067b-9649-4758-934b-459ca2bb383d-crio-socket\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.735984 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.735826    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.735984 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.735848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1124f153-b0c0-4132-88a3-c60068f9e2c3-config-volume\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.836877 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.836788    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.836877 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.836840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8c3349f-e8de-4229-93c5-8aa5499e3a72-cert\") pod \"ingress-canary-g7l9f\" (UID: \"c8c3349f-e8de-4229-93c5-8aa5499e3a72\") " pod="openshift-ingress-canary/ingress-canary-g7l9f"
Jul 29 10:29:13.837117 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.836884    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrgw\" (UniqueName: \"kubernetes.io/projected/090b067b-9649-4758-934b-459ca2bb383d-kube-api-access-8vrgw\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.837117 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.836914    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrv6\" (UniqueName: \"kubernetes.io/projected/1124f153-b0c0-4132-88a3-c60068f9e2c3-kube-api-access-4zrv6\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.837117 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:13.836995    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:13.837117 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:13.837074    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:14.337049605 +0000 UTC m=+36.565296150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:13.837117 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.837005    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/090b067b-9649-4758-934b-459ca2bb383d-data-volume\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.837378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.837159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.837378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.837195    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1124f153-b0c0-4132-88a3-c60068f9e2c3-tmp-dir\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.837378 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:13.837224    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:13.837378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.837224    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgrd\" (UniqueName: \"kubernetes.io/projected/c8c3349f-e8de-4229-93c5-8aa5499e3a72-kube-api-access-flgrd\") pod \"ingress-canary-g7l9f\" (UID: \"c8c3349f-e8de-4229-93c5-8aa5499e3a72\") " pod="openshift-ingress-canary/ingress-canary-g7l9f"
Jul 29 10:29:13.837378 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:13.837292    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls podName:1124f153-b0c0-4132-88a3-c60068f9e2c3 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:14.337275541 +0000 UTC m=+36.565522083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls") pod "dns-default-zmftl" (UID: "1124f153-b0c0-4132-88a3-c60068f9e2c3") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:13.837624 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.837433    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/090b067b-9649-4758-934b-459ca2bb383d-crio-socket\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.837624 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.837464    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.837624 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.837494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1124f153-b0c0-4132-88a3-c60068f9e2c3-config-volume\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.837624 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.837588    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1124f153-b0c0-4132-88a3-c60068f9e2c3-tmp-dir\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.837907 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.837819    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/090b067b-9649-4758-934b-459ca2bb383d-crio-socket\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.838041 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:13.837912    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:13.838041 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.838012    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/090b067b-9649-4758-934b-459ca2bb383d-data-volume\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.838207 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:13.838063    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:14.338043817 +0000 UTC m=+36.566290365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:13.838207 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.838085    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1124f153-b0c0-4132-88a3-c60068f9e2c3-config-volume\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.848565 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.848529    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrv6\" (UniqueName: \"kubernetes.io/projected/1124f153-b0c0-4132-88a3-c60068f9e2c3-kube-api-access-4zrv6\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:13.848718 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.848532    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrgw\" (UniqueName: \"kubernetes.io/projected/090b067b-9649-4758-934b-459ca2bb383d-kube-api-access-8vrgw\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:13.937967 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.937900    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgrd\" (UniqueName: \"kubernetes.io/projected/c8c3349f-e8de-4229-93c5-8aa5499e3a72-kube-api-access-flgrd\") pod \"ingress-canary-g7l9f\" (UID: \"c8c3349f-e8de-4229-93c5-8aa5499e3a72\") " pod="openshift-ingress-canary/ingress-canary-g7l9f"
Jul 29 10:29:13.938194 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.938142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8c3349f-e8de-4229-93c5-8aa5499e3a72-cert\") pod \"ingress-canary-g7l9f\" (UID: \"c8c3349f-e8de-4229-93c5-8aa5499e3a72\") " pod="openshift-ingress-canary/ingress-canary-g7l9f"
Jul 29 10:29:13.940703 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.940678    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8c3349f-e8de-4229-93c5-8aa5499e3a72-cert\") pod \"ingress-canary-g7l9f\" (UID: \"c8c3349f-e8de-4229-93c5-8aa5499e3a72\") " pod="openshift-ingress-canary/ingress-canary-g7l9f"
Jul 29 10:29:13.946619 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:13.946587    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgrd\" (UniqueName: \"kubernetes.io/projected/c8c3349f-e8de-4229-93c5-8aa5499e3a72-kube-api-access-flgrd\") pod \"ingress-canary-g7l9f\" (UID: \"c8c3349f-e8de-4229-93c5-8aa5499e3a72\") " pod="openshift-ingress-canary/ingress-canary-g7l9f"
Jul 29 10:29:14.027656 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.027629    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g7l9f"
Jul 29 10:29:14.215165 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.215132    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mzdqj"]
Jul 29 10:29:14.226585 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.226370    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g7l9f"]
Jul 29 10:29:14.226585 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.226496    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.227366 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.227335    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mzdqj"]
Jul 29 10:29:14.229768 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.229226    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dmrpd\""
Jul 29 10:29:14.229768 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:14.229289    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c3349f_e8de_4229_93c5_8aa5499e3a72.slice/crio-e957e0e0f29fca56e7017c582bb54dad765d9fe5535e2e576ff6a6fbc4937f85 WatchSource:0}: Error finding container e957e0e0f29fca56e7017c582bb54dad765d9fe5535e2e576ff6a6fbc4937f85: Status 404 returned error can't find the container with id e957e0e0f29fca56e7017c582bb54dad765d9fe5535e2e576ff6a6fbc4937f85
Jul 29 10:29:14.229768 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.229491    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 10:29:14.229768 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.229544    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 10:29:14.229768 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.229544    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 10:29:14.229768 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.229628    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 10:29:14.341554 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.341452    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:14.341554 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.341524    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.341571    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.341609    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9d4bf80d-43f5-4042-b9f7-ef702b055643-signing-cabundle\") pod \"service-ca-658d7b7c78-mzdqj\" (UID: \"9d4bf80d-43f5-4042-b9f7-ef702b055643\") " pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:14.341619    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.341635    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf2r2\" (UniqueName: \"kubernetes.io/projected/9d4bf80d-43f5-4042-b9f7-ef702b055643-kube-api-access-mf2r2\") pod \"service-ca-658d7b7c78-mzdqj\" (UID: \"9d4bf80d-43f5-4042-b9f7-ef702b055643\") " pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:14.341681    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:14.341701    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:14.341689    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls podName:1124f153-b0c0-4132-88a3-c60068f9e2c3 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:15.341669143 +0000 UTC m=+37.569915687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls") pod "dns-default-zmftl" (UID: "1124f153-b0c0-4132-88a3-c60068f9e2c3") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.341759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9d4bf80d-43f5-4042-b9f7-ef702b055643-signing-key\") pod \"service-ca-658d7b7c78-mzdqj\" (UID: \"9d4bf80d-43f5-4042-b9f7-ef702b055643\") " pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:14.341847    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:15.341833366 +0000 UTC m=+37.570079897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:14.342134 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:14.341859    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:15.341853509 +0000 UTC m=+37.570100036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:14.442842 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.442804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9d4bf80d-43f5-4042-b9f7-ef702b055643-signing-cabundle\") pod \"service-ca-658d7b7c78-mzdqj\" (UID: \"9d4bf80d-43f5-4042-b9f7-ef702b055643\") " pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.442842 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.442849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf2r2\" (UniqueName: \"kubernetes.io/projected/9d4bf80d-43f5-4042-b9f7-ef702b055643-kube-api-access-mf2r2\") pod \"service-ca-658d7b7c78-mzdqj\" (UID: \"9d4bf80d-43f5-4042-b9f7-ef702b055643\") " pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.443124 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.442946    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9d4bf80d-43f5-4042-b9f7-ef702b055643-signing-key\") pod \"service-ca-658d7b7c78-mzdqj\" (UID: \"9d4bf80d-43f5-4042-b9f7-ef702b055643\") " pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.443716 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.443695    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9d4bf80d-43f5-4042-b9f7-ef702b055643-signing-cabundle\") pod \"service-ca-658d7b7c78-mzdqj\" (UID: \"9d4bf80d-43f5-4042-b9f7-ef702b055643\") " pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.446576 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.446559    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9d4bf80d-43f5-4042-b9f7-ef702b055643-signing-key\") pod \"service-ca-658d7b7c78-mzdqj\" (UID: \"9d4bf80d-43f5-4042-b9f7-ef702b055643\") " pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.450135 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.450116    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf2r2\" (UniqueName: \"kubernetes.io/projected/9d4bf80d-43f5-4042-b9f7-ef702b055643-kube-api-access-mf2r2\") pod \"service-ca-658d7b7c78-mzdqj\" (UID: \"9d4bf80d-43f5-4042-b9f7-ef702b055643\") " pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.484301 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.484276    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xgkvk_a2f82d39-8f16-4606-a6d5-8f9ced950f85/node-ca/0.log"
Jul 29 10:29:14.542713 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.542665    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj"
Jul 29 10:29:14.551757 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.551719    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g7l9f" event={"ID":"c8c3349f-e8de-4229-93c5-8aa5499e3a72","Type":"ContainerStarted","Data":"e957e0e0f29fca56e7017c582bb54dad765d9fe5535e2e576ff6a6fbc4937f85"}
Jul 29 10:29:14.554619 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.554591    2572 generic.go:356] "Generic (PLEG): container finished" podID="56c584cc-8c25-427e-9209-d70836610d73" containerID="8332f7e9d85d24c06a46510de35d6267435fed41904019b63e4333eede3bdbe9" exitCode=0
Jul 29 10:29:14.554763 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.554632    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c4t8m" event={"ID":"56c584cc-8c25-427e-9209-d70836610d73","Type":"ContainerDied","Data":"8332f7e9d85d24c06a46510de35d6267435fed41904019b63e4333eede3bdbe9"}
Jul 29 10:29:14.697252 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:14.696994    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mzdqj"]
Jul 29 10:29:14.701843 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:14.701811    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4bf80d_43f5_4042_b9f7_ef702b055643.slice/crio-1b330cedf08d61a56cb35ac7f1109945bafb8869a8baa24bbb1d355a7ff3f0b1 WatchSource:0}: Error finding container 1b330cedf08d61a56cb35ac7f1109945bafb8869a8baa24bbb1d355a7ff3f0b1: Status 404 returned error can't find the container with id 1b330cedf08d61a56cb35ac7f1109945bafb8869a8baa24bbb1d355a7ff3f0b1
Jul 29 10:29:15.305516 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.305021    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:15.305516 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.305066    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:15.305516 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.305036    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:15.307672 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.307649    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:15.308318 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.308084    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:15.308318 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.308194    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:15.308472 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.308385    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:15.308615 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.308589    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wmbjw\""
Jul 29 10:29:15.308615 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.308599    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l8nkz\""
Jul 29 10:29:15.352372 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.351912    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:15.352372 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.352021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:15.352372 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.352067    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:15.352372 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:15.352260    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:15.352372 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:15.352312    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:15.352372 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:15.352314    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:15.352372 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:15.352325    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.352307011 +0000 UTC m=+39.580553543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:15.353164 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:15.352419    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.352401833 +0000 UTC m=+39.580648366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:15.353164 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:15.352437    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls podName:1124f153-b0c0-4132-88a3-c60068f9e2c3 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.352426847 +0000 UTC m=+39.580673377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls") pod "dns-default-zmftl" (UID: "1124f153-b0c0-4132-88a3-c60068f9e2c3") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:15.561853 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.561772    2572 generic.go:356] "Generic (PLEG): container finished" podID="56c584cc-8c25-427e-9209-d70836610d73" containerID="c493c9243f3a8b107bd1d24bee5620bc8686d308708f4ebec57eef371eeed0ed" exitCode=0
Jul 29 10:29:15.562018 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.561856    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c4t8m" event={"ID":"56c584cc-8c25-427e-9209-d70836610d73","Type":"ContainerDied","Data":"c493c9243f3a8b107bd1d24bee5620bc8686d308708f4ebec57eef371eeed0ed"}
Jul 29 10:29:15.564122 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:15.564089    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj" event={"ID":"9d4bf80d-43f5-4042-b9f7-ef702b055643","Type":"ContainerStarted","Data":"1b330cedf08d61a56cb35ac7f1109945bafb8869a8baa24bbb1d355a7ff3f0b1"}
Jul 29 10:29:16.571323 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:16.571281    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c4t8m" event={"ID":"56c584cc-8c25-427e-9209-d70836610d73","Type":"ContainerStarted","Data":"bb0c6de7c52ccf689e79d5a6ea73be0399d407ea89c7d0caeeac20922eb68689"}
Jul 29 10:29:16.597394 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:16.597333    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c4t8m" podStartSLOduration=8.777619675 podStartE2EDuration="38.597316714s" totalImagesPullingTime="29.819697039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:16.597141965 +0000 UTC m=+38.825388517" watchObservedRunningTime="2026-07-29 10:29:16.597316714 +0000 UTC m=+38.825563264"
Jul 29 10:29:17.371684 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:17.371645    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:17.371896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:17.371713    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:17.371896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:17.371748    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:17.371896 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:17.371787    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:17.371896 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:17.371856    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:17.372085 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:17.371860    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls podName:1124f153-b0c0-4132-88a3-c60068f9e2c3 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:21.371842263 +0000 UTC m=+43.600088795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls") pod "dns-default-zmftl" (UID: "1124f153-b0c0-4132-88a3-c60068f9e2c3") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:17.372085 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:17.371865    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:17.372085 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:17.371955    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:21.371918452 +0000 UTC m=+43.600164983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:17.372085 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:17.371987    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:21.371975536 +0000 UTC m=+43.600222084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:18.578108 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:18.578068    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g7l9f" event={"ID":"c8c3349f-e8de-4229-93c5-8aa5499e3a72","Type":"ContainerStarted","Data":"bddb22838ffb044f6a2f02500f2c2667372303b1e58f94beda522c0c466c3a9f"}
Jul 29 10:29:18.579688 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:18.579656    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj" event={"ID":"9d4bf80d-43f5-4042-b9f7-ef702b055643","Type":"ContainerStarted","Data":"999e9a80913bd0589c7eb78e8161eee9acfb2b4aa3328ee310349f08e7923f48"}
Jul 29 10:29:18.598547 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:18.598429    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g7l9f" podStartSLOduration=2.297319631 podStartE2EDuration="5.598408725s" totalImagesPullingTime="3.301089094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:18.598165595 +0000 UTC m=+40.826412146" watchObservedRunningTime="2026-07-29 10:29:18.598408725 +0000 UTC m=+40.826655276"
Jul 29 10:29:18.615355 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:18.615294    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-mzdqj" podStartSLOduration=1.780557007 podStartE2EDuration="4.615272538s" totalImagesPullingTime="2.834715531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:18.614301036 +0000 UTC m=+40.842547592" watchObservedRunningTime="2026-07-29 10:29:18.615272538 +0000 UTC m=+40.843519089"
Jul 29 10:29:21.402941 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:21.402880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:21.403323 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:21.402976    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:21.403323 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:21.403020    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:21.403323 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:21.403039    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:21.403323 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:21.403086    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:21.403323 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:21.403112    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:29.403096535 +0000 UTC m=+51.631343063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:21.403323 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:21.403112    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:21.403323 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:21.403126    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm podName:090b067b-9649-4758-934b-459ca2bb383d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:29.403120167 +0000 UTC m=+51.631366695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rwhkx" (UID: "090b067b-9649-4758-934b-459ca2bb383d") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:21.403323 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:29:21.403159    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls podName:1124f153-b0c0-4132-88a3-c60068f9e2c3 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:29.403145664 +0000 UTC m=+51.631392191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls") pod "dns-default-zmftl" (UID: "1124f153-b0c0-4132-88a3-c60068f9e2c3") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:29.462054 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.462004    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:29.462579 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.462070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:29.462579 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.462189    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:29.462706 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.462686    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/090b067b-9649-4758-934b-459ca2bb383d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:29.465229 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.465200    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/090b067b-9649-4758-934b-459ca2bb383d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rwhkx\" (UID: \"090b067b-9649-4758-934b-459ca2bb383d\") " pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:29.465334 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.465315    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1124f153-b0c0-4132-88a3-c60068f9e2c3-metrics-tls\") pod \"dns-default-zmftl\" (UID: \"1124f153-b0c0-4132-88a3-c60068f9e2c3\") " pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:29.527036 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.526995    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:29.533125 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.533098    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rwhkx"
Jul 29 10:29:29.704530 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.704496    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmftl"]
Jul 29 10:29:29.708120 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:29.708093    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1124f153_b0c0_4132_88a3_c60068f9e2c3.slice/crio-93d556e75c6e89a2576466c21cc2e35b3bf7d093e4f144a5c18ef44b3bd6df61 WatchSource:0}: Error finding container 93d556e75c6e89a2576466c21cc2e35b3bf7d093e4f144a5c18ef44b3bd6df61: Status 404 returned error can't find the container with id 93d556e75c6e89a2576466c21cc2e35b3bf7d093e4f144a5c18ef44b3bd6df61
Jul 29 10:29:29.719472 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:29.719350    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rwhkx"]
Jul 29 10:29:29.724313 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:29.724285    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090b067b_9649_4758_934b_459ca2bb383d.slice/crio-d9e331fbdfd9caf8464f64a3876395ecf374846ed0fb7dc3b17e6327bc390e17 WatchSource:0}: Error finding container d9e331fbdfd9caf8464f64a3876395ecf374846ed0fb7dc3b17e6327bc390e17: Status 404 returned error can't find the container with id d9e331fbdfd9caf8464f64a3876395ecf374846ed0fb7dc3b17e6327bc390e17
Jul 29 10:29:30.619436 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:30.617214    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmftl" event={"ID":"1124f153-b0c0-4132-88a3-c60068f9e2c3","Type":"ContainerStarted","Data":"93d556e75c6e89a2576466c21cc2e35b3bf7d093e4f144a5c18ef44b3bd6df61"}
Jul 29 10:29:30.619978 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:30.619617    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rwhkx" event={"ID":"090b067b-9649-4758-934b-459ca2bb383d","Type":"ContainerStarted","Data":"d9e331fbdfd9caf8464f64a3876395ecf374846ed0fb7dc3b17e6327bc390e17"}
Jul 29 10:29:32.627347 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:32.627301    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmftl" event={"ID":"1124f153-b0c0-4132-88a3-c60068f9e2c3","Type":"ContainerStarted","Data":"8247d162f5d8f77810ddede9422c30eafe6116f7b14f5177f50bfef78a0db53d"}
Jul 29 10:29:32.627811 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:32.627377    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmftl" event={"ID":"1124f153-b0c0-4132-88a3-c60068f9e2c3","Type":"ContainerStarted","Data":"ed65a7ecfe946295c2a88b9340d124fa7697387d3dbc9554d0269ec9ca5ad008"}
Jul 29 10:29:32.627811 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:32.627436    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:32.645962 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:32.645705    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zmftl" podStartSLOduration=17.436549005 podStartE2EDuration="19.645680999s" totalImagesPullingTime="2.209131994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:32.644228875 +0000 UTC m=+54.872475427" watchObservedRunningTime="2026-07-29 10:29:32.645680999 +0000 UTC m=+54.873927549"
Jul 29 10:29:35.638502 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:35.638310    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rwhkx" event={"ID":"090b067b-9649-4758-934b-459ca2bb383d","Type":"ContainerStarted","Data":"268eb828de586af6ac36ec7070f4d8e10e31077122acc39c2cbaffbb1ca6ec88"}
Jul 29 10:29:36.523464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.523427    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"]
Jul 29 10:29:36.529237 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.529210    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"]
Jul 29 10:29:36.529410 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.529339    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.531800 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.531665    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 10:29:36.531800 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.531677    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 10:29:36.532348 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.532173    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 10:29:36.532348 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.532236    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 10:29:36.534634 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.534490    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.535838 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.535814    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"]
Jul 29 10:29:36.536832 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.536226    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 29 10:29:36.536832 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.536716    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 29 10:29:36.537046 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.536916    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 29 10:29:36.537442 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.537263    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 29 10:29:36.546031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.546005    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"]
Jul 29 10:29:36.620967 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.620933    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.620967 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.620980    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd2b01fc-267a-4155-842d-62ede146d486-tmp\") pod \"klusterlet-addon-workmgr-c4c69b98d-skpxk\" (UID: \"fd2b01fc-267a-4155-842d-62ede146d486\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.621226 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.620998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.621226 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.621017    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-hub\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.621226 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.621032    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/518a81ad-aa18-4220-99e9-b22ec2464de8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.621226 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.621053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm27t\" (UniqueName: \"kubernetes.io/projected/518a81ad-aa18-4220-99e9-b22ec2464de8-kube-api-access-gm27t\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.621226 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.621194    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fd2b01fc-267a-4155-842d-62ede146d486-klusterlet-config\") pod \"klusterlet-addon-workmgr-c4c69b98d-skpxk\" (UID: \"fd2b01fc-267a-4155-842d-62ede146d486\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.621226 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.621216    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92tx\" (UniqueName: \"kubernetes.io/projected/fd2b01fc-267a-4155-842d-62ede146d486-kube-api-access-r92tx\") pod \"klusterlet-addon-workmgr-c4c69b98d-skpxk\" (UID: \"fd2b01fc-267a-4155-842d-62ede146d486\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.621437 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.621232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-ca\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.650744 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.650711    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5df59cb6-7ncb4"]
Jul 29 10:29:36.656156 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.656127    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.658967 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.658917    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 10:29:36.659172 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.659145    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 10:29:36.659303 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.659178    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 10:29:36.659372 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.659356    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6wkgt\""
Jul 29 10:29:36.668757 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.668729    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 10:29:36.682282 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.682248    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5df59cb6-7ncb4"]
Jul 29 10:29:36.722228 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722186    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.722422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d461338-d5de-41de-ba00-45dd33b27b02-trusted-ca\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.722422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722263    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd2b01fc-267a-4155-842d-62ede146d486-tmp\") pod \"klusterlet-addon-workmgr-c4c69b98d-skpxk\" (UID: \"fd2b01fc-267a-4155-842d-62ede146d486\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.722422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722285    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/518a81ad-aa18-4220-99e9-b22ec2464de8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.722422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722307    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d461338-d5de-41de-ba00-45dd33b27b02-registry-tls\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.722422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722384    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d461338-d5de-41de-ba00-45dd33b27b02-installation-pull-secrets\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.722422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722410    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d461338-d5de-41de-ba00-45dd33b27b02-bound-sa-token\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.722755 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722499    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fd2b01fc-267a-4155-842d-62ede146d486-klusterlet-config\") pod \"klusterlet-addon-workmgr-c4c69b98d-skpxk\" (UID: \"fd2b01fc-267a-4155-842d-62ede146d486\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.722755 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722535    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92tx\" (UniqueName: \"kubernetes.io/projected/fd2b01fc-267a-4155-842d-62ede146d486-kube-api-access-r92tx\") pod \"klusterlet-addon-workmgr-c4c69b98d-skpxk\" (UID: \"fd2b01fc-267a-4155-842d-62ede146d486\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.722755 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722571    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j72g\" (UniqueName: \"kubernetes.io/projected/6d461338-d5de-41de-ba00-45dd33b27b02-kube-api-access-6j72g\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.722755 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722615    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-ca\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.722755 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d461338-d5de-41de-ba00-45dd33b27b02-registry-certificates\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.722755 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722685    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd2b01fc-267a-4155-842d-62ede146d486-tmp\") pod \"klusterlet-addon-workmgr-c4c69b98d-skpxk\" (UID: \"fd2b01fc-267a-4155-842d-62ede146d486\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.722755 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722731    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d461338-d5de-41de-ba00-45dd33b27b02-image-registry-private-configuration\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.723091 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722763    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-hub\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.723091 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722788    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d461338-d5de-41de-ba00-45dd33b27b02-ca-trust-extracted\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.723091 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722838    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.723091 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.722866    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm27t\" (UniqueName: \"kubernetes.io/projected/518a81ad-aa18-4220-99e9-b22ec2464de8-kube-api-access-gm27t\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.723343 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.723119    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/518a81ad-aa18-4220-99e9-b22ec2464de8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.725714 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.725671    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-ca\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.725878 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.725772    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fd2b01fc-267a-4155-842d-62ede146d486-klusterlet-config\") pod \"klusterlet-addon-workmgr-c4c69b98d-skpxk\" (UID: \"fd2b01fc-267a-4155-842d-62ede146d486\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.725878 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.725856    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-hub\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.726092 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.726067    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.726377 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.726356    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/518a81ad-aa18-4220-99e9-b22ec2464de8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.730586 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.730540    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm27t\" (UniqueName: \"kubernetes.io/projected/518a81ad-aa18-4220-99e9-b22ec2464de8-kube-api-access-gm27t\") pod \"cluster-proxy-proxy-agent-6568fbb7bc-jtgzq\" (UID: \"518a81ad-aa18-4220-99e9-b22ec2464de8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.731440 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.731419    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92tx\" (UniqueName: \"kubernetes.io/projected/fd2b01fc-267a-4155-842d-62ede146d486-kube-api-access-r92tx\") pod \"klusterlet-addon-workmgr-c4c69b98d-skpxk\" (UID: \"fd2b01fc-267a-4155-842d-62ede146d486\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.823900 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.823802    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d461338-d5de-41de-ba00-45dd33b27b02-trusted-ca\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.823900 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.823852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d461338-d5de-41de-ba00-45dd33b27b02-registry-tls\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.823900 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.823900    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d461338-d5de-41de-ba00-45dd33b27b02-installation-pull-secrets\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.824215 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.823946    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d461338-d5de-41de-ba00-45dd33b27b02-bound-sa-token\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.824215 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.823983    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j72g\" (UniqueName: \"kubernetes.io/projected/6d461338-d5de-41de-ba00-45dd33b27b02-kube-api-access-6j72g\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.824215 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.824029    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d461338-d5de-41de-ba00-45dd33b27b02-registry-certificates\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.824215 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.824072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d461338-d5de-41de-ba00-45dd33b27b02-image-registry-private-configuration\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.824215 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.824097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d461338-d5de-41de-ba00-45dd33b27b02-ca-trust-extracted\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.824841 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.824814    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d461338-d5de-41de-ba00-45dd33b27b02-ca-trust-extracted\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.824968 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.824894    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d461338-d5de-41de-ba00-45dd33b27b02-registry-certificates\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.825114 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.825063    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d461338-d5de-41de-ba00-45dd33b27b02-trusted-ca\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.827021 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.826990    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d461338-d5de-41de-ba00-45dd33b27b02-registry-tls\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.827120 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.827099    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d461338-d5de-41de-ba00-45dd33b27b02-installation-pull-secrets\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.827320 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.827296    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d461338-d5de-41de-ba00-45dd33b27b02-image-registry-private-configuration\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.834962 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.834886    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j72g\" (UniqueName: \"kubernetes.io/projected/6d461338-d5de-41de-ba00-45dd33b27b02-kube-api-access-6j72g\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.835270 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.835242    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d461338-d5de-41de-ba00-45dd33b27b02-bound-sa-token\") pod \"image-registry-5df59cb6-7ncb4\" (UID: \"6d461338-d5de-41de-ba00-45dd33b27b02\") " pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:36.848382 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.848340    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:36.863279 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.863231    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"
Jul 29 10:29:36.967651 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:36.967612    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:39.049705 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.049672    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq"]
Jul 29 10:29:39.054957 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:39.054906    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod518a81ad_aa18_4220_99e9_b22ec2464de8.slice/crio-9a706c066284b918477a201e48dc111beb96f0683866de303e988ad171f860fe WatchSource:0}: Error finding container 9a706c066284b918477a201e48dc111beb96f0683866de303e988ad171f860fe: Status 404 returned error can't find the container with id 9a706c066284b918477a201e48dc111beb96f0683866de303e988ad171f860fe
Jul 29 10:29:39.075280 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.075249    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5df59cb6-7ncb4"]
Jul 29 10:29:39.078480 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:39.078457    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d461338_d5de_41de_ba00_45dd33b27b02.slice/crio-a152a094553d7e261d826e5310eb3d837ff358d8ebc211f5699e1401f12633de WatchSource:0}: Error finding container a152a094553d7e261d826e5310eb3d837ff358d8ebc211f5699e1401f12633de: Status 404 returned error can't find the container with id a152a094553d7e261d826e5310eb3d837ff358d8ebc211f5699e1401f12633de
Jul 29 10:29:39.088090 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.088055    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"]
Jul 29 10:29:39.091621 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:39.091588    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2b01fc_267a_4155_842d_62ede146d486.slice/crio-21d76f9f86a90709172ce6c979c0761c527e88e5703876c8bd23533340daa741 WatchSource:0}: Error finding container 21d76f9f86a90709172ce6c979c0761c527e88e5703876c8bd23533340daa741: Status 404 returned error can't find the container with id 21d76f9f86a90709172ce6c979c0761c527e88e5703876c8bd23533340daa741
Jul 29 10:29:39.552388 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.552349    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2hvgf"
Jul 29 10:29:39.652599 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.652560    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5df59cb6-7ncb4" event={"ID":"6d461338-d5de-41de-ba00-45dd33b27b02","Type":"ContainerStarted","Data":"f459507a1017584349e83ece6f79f614f08c39497d2b70f7dd8f18b31cc7efce"}
Jul 29 10:29:39.652817 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.652616    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5df59cb6-7ncb4" event={"ID":"6d461338-d5de-41de-ba00-45dd33b27b02","Type":"ContainerStarted","Data":"a152a094553d7e261d826e5310eb3d837ff358d8ebc211f5699e1401f12633de"}
Jul 29 10:29:39.652817 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.652732    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:29:39.653960 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.653909    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk" event={"ID":"fd2b01fc-267a-4155-842d-62ede146d486","Type":"ContainerStarted","Data":"21d76f9f86a90709172ce6c979c0761c527e88e5703876c8bd23533340daa741"}
Jul 29 10:29:39.656048 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.655981    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rwhkx" event={"ID":"090b067b-9649-4758-934b-459ca2bb383d","Type":"ContainerStarted","Data":"7a488301767f7513403f2f1537750beaaa01570fc7a48c8b285be0908fa49e3c"}
Jul 29 10:29:39.657183 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.657153    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq" event={"ID":"518a81ad-aa18-4220-99e9-b22ec2464de8","Type":"ContainerStarted","Data":"9a706c066284b918477a201e48dc111beb96f0683866de303e988ad171f860fe"}
Jul 29 10:29:39.678406 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:39.678313    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5df59cb6-7ncb4" podStartSLOduration=3.67828893 podStartE2EDuration="3.67828893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:39.676907416 +0000 UTC m=+61.905153967" watchObservedRunningTime="2026-07-29 10:29:39.67828893 +0000 UTC m=+61.906535466"
Jul 29 10:29:42.632762 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:42.632646    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zmftl"
Jul 29 10:29:43.989631 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:43.989591    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:43.991796 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:43.991768    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:44.002989 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.002954    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2f0ed09-a52b-4305-8fa6-435794cc0ff3-original-pull-secret\") pod \"global-pull-secret-syncer-fcpjf\" (UID: \"b2f0ed09-a52b-4305-8fa6-435794cc0ff3\") " pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:44.090566 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.090519    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:44.092797 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.092761    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:44.103992 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.103963    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be3cb65-5879-412b-b438-99e55616cf3e-metrics-certs\") pod \"network-metrics-daemon-l8kkx\" (UID: \"7be3cb65-5879-412b-b438-99e55616cf3e\") " pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:44.120444 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.120412    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l8nkz\""
Jul 29 10:29:44.127346 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.127310    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fcpjf"
Jul 29 10:29:44.128364 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.128329    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l8kkx"
Jul 29 10:29:44.191744 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.191711    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:44.194270 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.194242    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:44.204338 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.204299    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:44.215877 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.215853    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f969l\" (UniqueName: \"kubernetes.io/projected/084f30e4-3825-48a8-96ca-c5ebc0c10b5a-kube-api-access-f969l\") pod \"network-check-target-wjnft\" (UID: \"084f30e4-3825-48a8-96ca-c5ebc0c10b5a\") " pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:44.436791 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.436755    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wmbjw\""
Jul 29 10:29:44.445465 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:44.445424    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:46.679566 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.679447    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fcpjf"]
Jul 29 10:29:46.685186 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:46.685143    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f0ed09_a52b_4305_8fa6_435794cc0ff3.slice/crio-db898eb920fdd2060365c9779f56d4df21535f6a0d8c4be30ac0b893987fb2d5 WatchSource:0}: Error finding container db898eb920fdd2060365c9779f56d4df21535f6a0d8c4be30ac0b893987fb2d5: Status 404 returned error can't find the container with id db898eb920fdd2060365c9779f56d4df21535f6a0d8c4be30ac0b893987fb2d5
Jul 29 10:29:46.697464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.697327    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rwhkx" event={"ID":"090b067b-9649-4758-934b-459ca2bb383d","Type":"ContainerStarted","Data":"7e709b8a928e2afe22ef8867182b9fc7ceca5888c52057379a88e01e48d8c3e1"}
Jul 29 10:29:46.699112 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.699076    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l8kkx"]
Jul 29 10:29:46.699668 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.699344    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq" event={"ID":"518a81ad-aa18-4220-99e9-b22ec2464de8","Type":"ContainerStarted","Data":"1b71103209cae49295dfef6da44030212d5be8e09c056a09e18754b6ca98087f"}
Jul 29 10:29:46.702685 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.702569    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk" event={"ID":"fd2b01fc-267a-4155-842d-62ede146d486","Type":"ContainerStarted","Data":"4b1a66d171ed60d4a7c5c312dcc2f80a7d2a56a13b6b2a36def79e2078080581"}
Jul 29 10:29:46.703195 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.703174    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:46.705042 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.705020    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk"
Jul 29 10:29:46.705365 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:46.705327    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be3cb65_5879_412b_b438_99e55616cf3e.slice/crio-58b8b13a036f26fe4d2da1e8305ea950da4fba4dc6e65ffa9f3f17c848f6e66c WatchSource:0}: Error finding container 58b8b13a036f26fe4d2da1e8305ea950da4fba4dc6e65ffa9f3f17c848f6e66c: Status 404 returned error can't find the container with id 58b8b13a036f26fe4d2da1e8305ea950da4fba4dc6e65ffa9f3f17c848f6e66c
Jul 29 10:29:46.716382 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.716319    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rwhkx" podStartSLOduration=17.080025272 podStartE2EDuration="33.716298891s" totalImagesPullingTime="16.636273619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:46.715155808 +0000 UTC m=+68.943402359" watchObservedRunningTime="2026-07-29 10:29:46.716298891 +0000 UTC m=+68.944545467"
Jul 29 10:29:46.724419 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.724387    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wjnft"]
Jul 29 10:29:46.728333 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:46.728294    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084f30e4_3825_48a8_96ca_c5ebc0c10b5a.slice/crio-60771ed31e45a08b4ad7077539f9d0ac4ec003fe6f87f2589b08e1133c4ec454 WatchSource:0}: Error finding container 60771ed31e45a08b4ad7077539f9d0ac4ec003fe6f87f2589b08e1133c4ec454: Status 404 returned error can't find the container with id 60771ed31e45a08b4ad7077539f9d0ac4ec003fe6f87f2589b08e1133c4ec454
Jul 29 10:29:46.730608 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:46.730509    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4c69b98d-skpxk" podStartSLOduration=3.31151874 podStartE2EDuration="10.730488333s" totalImagesPullingTime="7.418969593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:46.729330095 +0000 UTC m=+68.957576646" watchObservedRunningTime="2026-07-29 10:29:46.730488333 +0000 UTC m=+68.958734885"
Jul 29 10:29:47.711036 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:47.711001    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq" event={"ID":"518a81ad-aa18-4220-99e9-b22ec2464de8","Type":"ContainerStarted","Data":"9ae60d85aa90c19138a7a8c212ea1b1420fe2038f4e45113fb0039f74b7704a8"}
Jul 29 10:29:47.711499 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:47.711049    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq" event={"ID":"518a81ad-aa18-4220-99e9-b22ec2464de8","Type":"ContainerStarted","Data":"b8d3f300d1658464cf253f1064b0c1c955921024ab3f8eebce971932e1204acb"}
Jul 29 10:29:47.713255 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:47.713124    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wjnft" event={"ID":"084f30e4-3825-48a8-96ca-c5ebc0c10b5a","Type":"ContainerStarted","Data":"60771ed31e45a08b4ad7077539f9d0ac4ec003fe6f87f2589b08e1133c4ec454"}
Jul 29 10:29:47.715621 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:47.715554    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l8kkx" event={"ID":"7be3cb65-5879-412b-b438-99e55616cf3e","Type":"ContainerStarted","Data":"58b8b13a036f26fe4d2da1e8305ea950da4fba4dc6e65ffa9f3f17c848f6e66c"}
Jul 29 10:29:47.717749 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:47.717530    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fcpjf" event={"ID":"b2f0ed09-a52b-4305-8fa6-435794cc0ff3","Type":"ContainerStarted","Data":"db898eb920fdd2060365c9779f56d4df21535f6a0d8c4be30ac0b893987fb2d5"}
Jul 29 10:29:47.733824 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:47.733756    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6568fbb7bc-jtgzq" podStartSLOduration=4.271782692 podStartE2EDuration="11.733735228s" totalImagesPullingTime="7.461952536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:47.733326258 +0000 UTC m=+69.961572810" watchObservedRunningTime="2026-07-29 10:29:47.733735228 +0000 UTC m=+69.961981777"
Jul 29 10:29:48.725659 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:48.725616    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l8kkx" event={"ID":"7be3cb65-5879-412b-b438-99e55616cf3e","Type":"ContainerStarted","Data":"2e99f3a1fde02236cfe5e4444f73dc40b85a39b9d7744e53c03615296b246dd1"}
Jul 29 10:29:48.726246 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:48.725691    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l8kkx" event={"ID":"7be3cb65-5879-412b-b438-99e55616cf3e","Type":"ContainerStarted","Data":"5c069a3da2c7821fb0b7353f7b441d222938ea5af9f5181733f0ecd5b15e7550"}
Jul 29 10:29:48.743543 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:48.743468    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l8kkx" podStartSLOduration=69.688718781 podStartE2EDuration="1m10.743440502s" totalImagesPullingTime="1.054721721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:48.743210485 +0000 UTC m=+70.971457036" watchObservedRunningTime="2026-07-29 10:29:48.743440502 +0000 UTC m=+70.971687056"
Jul 29 10:29:52.741695 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:52.741652    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wjnft" event={"ID":"084f30e4-3825-48a8-96ca-c5ebc0c10b5a","Type":"ContainerStarted","Data":"25c184854990e494f0ad3e466c0006dce3614cb3a197cae2cf983cd248338afb"}
Jul 29 10:29:52.742184 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:52.741776    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:29:52.743222 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:52.743199    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fcpjf" event={"ID":"b2f0ed09-a52b-4305-8fa6-435794cc0ff3","Type":"ContainerStarted","Data":"baca019d463296cc963fc3233d6891d8e002660751ac7992d4ac0fb0399036f7"}
Jul 29 10:29:52.760626 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:52.760566    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wjnft" podStartSLOduration=69.777633629 podStartE2EDuration="1m14.760547703s" totalImagesPullingTime="4.982914074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:52.759821283 +0000 UTC m=+74.988067833" watchObservedRunningTime="2026-07-29 10:29:52.760547703 +0000 UTC m=+74.988794253"
Jul 29 10:29:52.773893 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:52.773826    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fcpjf" podStartSLOduration=69.737358022 podStartE2EDuration="1m14.773807301s" totalImagesPullingTime="5.036449279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:52.773768635 +0000 UTC m=+75.002015185" watchObservedRunningTime="2026-07-29 10:29:52.773807301 +0000 UTC m=+75.002053841"
Jul 29 10:29:56.921156 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.921113    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gwdgm"]
Jul 29 10:29:56.924899 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.924872    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:56.927182 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.927154    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 10:29:56.928780 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.928756    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 10:29:56.928917 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.928756    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 10:29:56.929464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.929445    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 10:29:56.929574 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.929450    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 10:29:56.929718 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.929701    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 10:29:56.929806 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.929740    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dcmhg\""
Jul 29 10:29:56.996297 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.996259    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f113b545-977d-4f0e-93b0-8bdd345dbfe5-sys\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:56.996297 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.996307    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-textfile\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:56.996549 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.996353    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f113b545-977d-4f0e-93b0-8bdd345dbfe5-metrics-client-ca\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:56.996549 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.996382    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f113b545-977d-4f0e-93b0-8bdd345dbfe5-root\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:56.996549 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.996406    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476xb\" (UniqueName: \"kubernetes.io/projected/f113b545-977d-4f0e-93b0-8bdd345dbfe5-kube-api-access-476xb\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:56.996549 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.996422    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:56.996549 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.996492    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-accelerators-collector-config\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:56.996549 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.996519    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-tls\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:56.996549 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:56.996538    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-wtmp\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.097676 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-textfile\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.097884 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097698    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f113b545-977d-4f0e-93b0-8bdd345dbfe5-metrics-client-ca\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.097884 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f113b545-977d-4f0e-93b0-8bdd345dbfe5-root\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.097884 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097760    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476xb\" (UniqueName: \"kubernetes.io/projected/f113b545-977d-4f0e-93b0-8bdd345dbfe5-kube-api-access-476xb\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.097884 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097786    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.097884 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-accelerators-collector-config\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.097884 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-tls\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.097884 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097845    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f113b545-977d-4f0e-93b0-8bdd345dbfe5-root\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.097884 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097872    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-wtmp\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.098316 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.097961    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f113b545-977d-4f0e-93b0-8bdd345dbfe5-sys\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.098316 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.098013    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-wtmp\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.098316 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.098083    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f113b545-977d-4f0e-93b0-8bdd345dbfe5-sys\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.098316 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.098090    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-textfile\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.098500 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.098387    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f113b545-977d-4f0e-93b0-8bdd345dbfe5-metrics-client-ca\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.098543 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.098524    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-accelerators-collector-config\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.100746 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.100721    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.101000 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.100978    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f113b545-977d-4f0e-93b0-8bdd345dbfe5-node-exporter-tls\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.138114 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.138079    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-476xb\" (UniqueName: \"kubernetes.io/projected/f113b545-977d-4f0e-93b0-8bdd345dbfe5-kube-api-access-476xb\") pod \"node-exporter-gwdgm\" (UID: \"f113b545-977d-4f0e-93b0-8bdd345dbfe5\") " pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.236444 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.236350    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gwdgm"
Jul 29 10:29:57.248573 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:29:57.248536    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf113b545_977d_4f0e_93b0_8bdd345dbfe5.slice/crio-0f72d3c2f620e83d48ebba195311f344f265f68fb1be0b38765107e8d5ecc3a9 WatchSource:0}: Error finding container 0f72d3c2f620e83d48ebba195311f344f265f68fb1be0b38765107e8d5ecc3a9: Status 404 returned error can't find the container with id 0f72d3c2f620e83d48ebba195311f344f265f68fb1be0b38765107e8d5ecc3a9
Jul 29 10:29:57.766494 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:57.766459    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwdgm" event={"ID":"f113b545-977d-4f0e-93b0-8bdd345dbfe5","Type":"ContainerStarted","Data":"0f72d3c2f620e83d48ebba195311f344f265f68fb1be0b38765107e8d5ecc3a9"}
Jul 29 10:29:58.771834 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:58.771794    2572 generic.go:356] "Generic (PLEG): container finished" podID="f113b545-977d-4f0e-93b0-8bdd345dbfe5" containerID="cb4107ba3352146e8a2b73f3cd637c900686abd08f6a25552dc2e9494ae72e08" exitCode=0
Jul 29 10:29:58.772246 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:58.771865    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwdgm" event={"ID":"f113b545-977d-4f0e-93b0-8bdd345dbfe5","Type":"ContainerDied","Data":"cb4107ba3352146e8a2b73f3cd637c900686abd08f6a25552dc2e9494ae72e08"}
Jul 29 10:29:59.778135 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:59.778084    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwdgm" event={"ID":"f113b545-977d-4f0e-93b0-8bdd345dbfe5","Type":"ContainerStarted","Data":"b0cc3873656e08d09535009b3219bfc6f7caaecb54d552ef7fdf9b3bf3876587"}
Jul 29 10:29:59.778559 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:59.778191    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwdgm" event={"ID":"f113b545-977d-4f0e-93b0-8bdd345dbfe5","Type":"ContainerStarted","Data":"542c769edc3f0ad8a77914569e4879bc9c27611675d0c58770e3f80391363f88"}
Jul 29 10:29:59.798938 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:29:59.798843    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gwdgm" podStartSLOduration=2.855645841 podStartE2EDuration="3.79881945s" totalImagesPullingTime="872.133656ms" totalInitContainerRuntime="71.039953ms" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:59.796742732 +0000 UTC m=+82.024989282" watchObservedRunningTime="2026-07-29 10:29:59.79881945 +0000 UTC m=+82.027066002"
Jul 29 10:30:00.667198 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:30:00.667165    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5df59cb6-7ncb4"
Jul 29 10:30:23.750072 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:30:23.750040    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wjnft"
Jul 29 10:33:38.155206 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:33:38.155179    2572 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 10:34:09.760603 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.760565    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-cmzvx"]
Jul 29 10:34:09.762562 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.762545    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:34:09.764509 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.764483    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 29 10:34:09.764509 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.764506    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 10:34:09.765019 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.764999    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xd789\""
Jul 29 10:34:09.765134 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.765005    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 10:34:09.773906 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.773880    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-cmzvx"]
Jul 29 10:34:09.869785 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.869748    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfq26\" (UniqueName: \"kubernetes.io/projected/d9980c2e-b5cc-4d41-9481-52814816f0ed-kube-api-access-gfq26\") pod \"seaweedfs-64568bcd49-cmzvx\" (UID: \"d9980c2e-b5cc-4d41-9481-52814816f0ed\") " pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:34:09.869785 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.869793    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9980c2e-b5cc-4d41-9481-52814816f0ed-data\") pod \"seaweedfs-64568bcd49-cmzvx\" (UID: \"d9980c2e-b5cc-4d41-9481-52814816f0ed\") " pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:34:09.971153 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.971100    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfq26\" (UniqueName: \"kubernetes.io/projected/d9980c2e-b5cc-4d41-9481-52814816f0ed-kube-api-access-gfq26\") pod \"seaweedfs-64568bcd49-cmzvx\" (UID: \"d9980c2e-b5cc-4d41-9481-52814816f0ed\") " pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:34:09.971360 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.971164    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9980c2e-b5cc-4d41-9481-52814816f0ed-data\") pod \"seaweedfs-64568bcd49-cmzvx\" (UID: \"d9980c2e-b5cc-4d41-9481-52814816f0ed\") " pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:34:09.971568 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.971547    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9980c2e-b5cc-4d41-9481-52814816f0ed-data\") pod \"seaweedfs-64568bcd49-cmzvx\" (UID: \"d9980c2e-b5cc-4d41-9481-52814816f0ed\") " pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:34:09.982187 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:09.982132    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfq26\" (UniqueName: \"kubernetes.io/projected/d9980c2e-b5cc-4d41-9481-52814816f0ed-kube-api-access-gfq26\") pod \"seaweedfs-64568bcd49-cmzvx\" (UID: \"d9980c2e-b5cc-4d41-9481-52814816f0ed\") " pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:34:10.073556 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:10.073466    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:34:10.205527 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:10.205493    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-cmzvx"]
Jul 29 10:34:10.209147 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:34:10.209122    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9980c2e_b5cc_4d41_9481_52814816f0ed.slice/crio-e2e646a95ec57039de86824e8fd0c5cec58c89a693bed4e6c019e77056582ff8 WatchSource:0}: Error finding container e2e646a95ec57039de86824e8fd0c5cec58c89a693bed4e6c019e77056582ff8: Status 404 returned error can't find the container with id e2e646a95ec57039de86824e8fd0c5cec58c89a693bed4e6c019e77056582ff8
Jul 29 10:34:10.210359 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:10.210341    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:34:10.627704 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:10.627655    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-cmzvx" event={"ID":"d9980c2e-b5cc-4d41-9481-52814816f0ed","Type":"ContainerStarted","Data":"e2e646a95ec57039de86824e8fd0c5cec58c89a693bed4e6c019e77056582ff8"}
Jul 29 10:34:13.639495 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:13.639457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-cmzvx" event={"ID":"d9980c2e-b5cc-4d41-9481-52814816f0ed","Type":"ContainerStarted","Data":"ac1810e1ee4eb39efaba3968f506c8d995a18b15468e020a3767e0da436ae2b1"}
Jul 29 10:34:13.639951 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:13.639583    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:34:13.653842 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:13.653791    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-cmzvx" podStartSLOduration=1.666717349 podStartE2EDuration="4.65376791s" totalImagesPullingTime="2.987050561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:34:13.653025117 +0000 UTC m=+335.881271693" watchObservedRunningTime="2026-07-29 10:34:13.65376791 +0000 UTC m=+335.882014460"
Jul 29 10:34:19.644424 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:34:19.644387    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-cmzvx"
Jul 29 10:35:26.620790 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.620748    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-w9cz2"]
Jul 29 10:35:26.624148 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.624124    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:26.626606 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.626585    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-2j92x\""
Jul 29 10:35:26.626733 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.626602    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 29 10:35:26.629226 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.629199    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-6xvjw"]
Jul 29 10:35:26.632998 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.632975    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-w9cz2"]
Jul 29 10:35:26.633125 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.633073    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:26.635313 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.635288    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-qc42s\""
Jul 29 10:35:26.635433 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.635381    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 29 10:35:26.641079 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.641052    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-6xvjw"]
Jul 29 10:35:26.722257 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.722210    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjsfs\" (UniqueName: \"kubernetes.io/projected/ae28219a-e857-41fb-96cc-fd277c228c04-kube-api-access-sjsfs\") pod \"odh-model-controller-75d9cdbddf-6xvjw\" (UID: \"ae28219a-e857-41fb-96cc-fd277c228c04\") " pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:26.722257 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.722259    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82tl\" (UniqueName: \"kubernetes.io/projected/1868c5d5-3cfe-4553-8b21-78a5eadb082b-kube-api-access-c82tl\") pod \"model-serving-api-7b954ffc95-w9cz2\" (UID: \"1868c5d5-3cfe-4553-8b21-78a5eadb082b\") " pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:26.722501 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.722298    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae28219a-e857-41fb-96cc-fd277c228c04-cert\") pod \"odh-model-controller-75d9cdbddf-6xvjw\" (UID: \"ae28219a-e857-41fb-96cc-fd277c228c04\") " pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:26.722501 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.722349    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1868c5d5-3cfe-4553-8b21-78a5eadb082b-tls-certs\") pod \"model-serving-api-7b954ffc95-w9cz2\" (UID: \"1868c5d5-3cfe-4553-8b21-78a5eadb082b\") " pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:26.823335 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.823293    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae28219a-e857-41fb-96cc-fd277c228c04-cert\") pod \"odh-model-controller-75d9cdbddf-6xvjw\" (UID: \"ae28219a-e857-41fb-96cc-fd277c228c04\") " pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:26.823335 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.823338    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1868c5d5-3cfe-4553-8b21-78a5eadb082b-tls-certs\") pod \"model-serving-api-7b954ffc95-w9cz2\" (UID: \"1868c5d5-3cfe-4553-8b21-78a5eadb082b\") " pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:26.823577 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.823375    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjsfs\" (UniqueName: \"kubernetes.io/projected/ae28219a-e857-41fb-96cc-fd277c228c04-kube-api-access-sjsfs\") pod \"odh-model-controller-75d9cdbddf-6xvjw\" (UID: \"ae28219a-e857-41fb-96cc-fd277c228c04\") " pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:26.823577 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.823395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82tl\" (UniqueName: \"kubernetes.io/projected/1868c5d5-3cfe-4553-8b21-78a5eadb082b-kube-api-access-c82tl\") pod \"model-serving-api-7b954ffc95-w9cz2\" (UID: \"1868c5d5-3cfe-4553-8b21-78a5eadb082b\") " pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:26.823577 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:35:26.823456    2572 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 29 10:35:26.823577 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:35:26.823551    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae28219a-e857-41fb-96cc-fd277c228c04-cert podName:ae28219a-e857-41fb-96cc-fd277c228c04 nodeName:}" failed. No retries permitted until 2026-07-29 10:35:27.323526681 +0000 UTC m=+409.551773230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ae28219a-e857-41fb-96cc-fd277c228c04-cert") pod "odh-model-controller-75d9cdbddf-6xvjw" (UID: "ae28219a-e857-41fb-96cc-fd277c228c04") : secret "odh-model-controller-webhook-cert" not found
Jul 29 10:35:26.825952 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.825916    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1868c5d5-3cfe-4553-8b21-78a5eadb082b-tls-certs\") pod \"model-serving-api-7b954ffc95-w9cz2\" (UID: \"1868c5d5-3cfe-4553-8b21-78a5eadb082b\") " pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:26.845211 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.845178    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjsfs\" (UniqueName: \"kubernetes.io/projected/ae28219a-e857-41fb-96cc-fd277c228c04-kube-api-access-sjsfs\") pod \"odh-model-controller-75d9cdbddf-6xvjw\" (UID: \"ae28219a-e857-41fb-96cc-fd277c228c04\") " pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:26.845353 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.845275    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82tl\" (UniqueName: \"kubernetes.io/projected/1868c5d5-3cfe-4553-8b21-78a5eadb082b-kube-api-access-c82tl\") pod \"model-serving-api-7b954ffc95-w9cz2\" (UID: \"1868c5d5-3cfe-4553-8b21-78a5eadb082b\") " pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:26.935596 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:26.935494    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:27.072738 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:27.072704    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-w9cz2"]
Jul 29 10:35:27.076158 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:35:27.076126    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1868c5d5_3cfe_4553_8b21_78a5eadb082b.slice/crio-18479695db171d77080600e26f3b62534972c1125abc883db320c4693d4bcc9c WatchSource:0}: Error finding container 18479695db171d77080600e26f3b62534972c1125abc883db320c4693d4bcc9c: Status 404 returned error can't find the container with id 18479695db171d77080600e26f3b62534972c1125abc883db320c4693d4bcc9c
Jul 29 10:35:27.327876 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:27.327770    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae28219a-e857-41fb-96cc-fd277c228c04-cert\") pod \"odh-model-controller-75d9cdbddf-6xvjw\" (UID: \"ae28219a-e857-41fb-96cc-fd277c228c04\") " pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:27.330227 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:27.330194    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae28219a-e857-41fb-96cc-fd277c228c04-cert\") pod \"odh-model-controller-75d9cdbddf-6xvjw\" (UID: \"ae28219a-e857-41fb-96cc-fd277c228c04\") " pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:27.544600 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:27.544568    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:27.683041 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:27.682992    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-6xvjw"]
Jul 29 10:35:27.687594 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:35:27.687545    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae28219a_e857_41fb_96cc_fd277c228c04.slice/crio-352f9a8cf22195058a5cdf48dbf9546d26e800e53a98962a0683b145e2218d3a WatchSource:0}: Error finding container 352f9a8cf22195058a5cdf48dbf9546d26e800e53a98962a0683b145e2218d3a: Status 404 returned error can't find the container with id 352f9a8cf22195058a5cdf48dbf9546d26e800e53a98962a0683b145e2218d3a
Jul 29 10:35:27.887004 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:27.886909    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-6xvjw" event={"ID":"ae28219a-e857-41fb-96cc-fd277c228c04","Type":"ContainerStarted","Data":"352f9a8cf22195058a5cdf48dbf9546d26e800e53a98962a0683b145e2218d3a"}
Jul 29 10:35:27.888378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:27.888339    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-w9cz2" event={"ID":"1868c5d5-3cfe-4553-8b21-78a5eadb082b","Type":"ContainerStarted","Data":"18479695db171d77080600e26f3b62534972c1125abc883db320c4693d4bcc9c"}
Jul 29 10:35:29.899186 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:29.899052    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-w9cz2" event={"ID":"1868c5d5-3cfe-4553-8b21-78a5eadb082b","Type":"ContainerStarted","Data":"93f3728eac29b17b0aa94acddb3ab211d283d2b8d7c29b77265bc2d93356ac78"}
Jul 29 10:35:29.899186 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:29.899155    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:29.919170 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:29.919103    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-w9cz2" podStartSLOduration=1.634403423 podStartE2EDuration="3.91907855s" totalImagesPullingTime="2.284675127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:35:29.916525309 +0000 UTC m=+412.144771867" watchObservedRunningTime="2026-07-29 10:35:29.91907855 +0000 UTC m=+412.147325129"
Jul 29 10:35:31.908859 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:31.908818    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-6xvjw" event={"ID":"ae28219a-e857-41fb-96cc-fd277c228c04","Type":"ContainerStarted","Data":"a269ce31b149a44d9ae0ed6d68cdd20a630b37687314812c9c1b516d05a79cbe"}
Jul 29 10:35:31.909326 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:31.908980    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:35:31.926666 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:31.926606    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-6xvjw" podStartSLOduration=2.194465576 podStartE2EDuration="5.926588092s" totalImagesPullingTime="3.732122516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:35:31.924935451 +0000 UTC m=+414.153181993" watchObservedRunningTime="2026-07-29 10:35:31.926588092 +0000 UTC m=+414.154834641"
Jul 29 10:35:40.908025 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:40.907993    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-w9cz2"
Jul 29 10:35:42.916239 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:35:42.916207    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-6xvjw"
Jul 29 10:37:14.832444 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.832400    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"]
Jul 29 10:37:14.836007 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.835982    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:14.838326 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.838292    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f0182-kube-rbac-proxy-sar-config\""
Jul 29 10:37:14.838478 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.838292    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:37:14.838478 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.838361    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fqxzr\""
Jul 29 10:37:14.838598 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.838323    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f0182-predictor-serving-cert\""
Jul 29 10:37:14.838729 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.838714    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:37:14.847061 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.847034    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"]
Jul 29 10:37:14.924058 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.924018    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb7ffb2-19af-4e74-b44b-1dbb70855951-proxy-tls\") pod \"success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") " pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:14.924058 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.924057    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") " pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:14.924349 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:14.924086    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6gf\" (UniqueName: \"kubernetes.io/projected/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-api-access-rn6gf\") pod \"success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") " pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:15.024917 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.024873    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6gf\" (UniqueName: \"kubernetes.io/projected/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-api-access-rn6gf\") pod \"success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") " pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:15.025127 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.024972    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb7ffb2-19af-4e74-b44b-1dbb70855951-proxy-tls\") pod \"success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") " pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:15.025127 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.025000    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") " pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:15.025707 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.025687    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") " pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:15.027809 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.027771    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb7ffb2-19af-4e74-b44b-1dbb70855951-proxy-tls\") pod \"success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") " pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:15.036116 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.036083    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6gf\" (UniqueName: \"kubernetes.io/projected/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-api-access-rn6gf\") pod \"success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") " pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:15.139830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.139793    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"]
Jul 29 10:37:15.143341 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.143320    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.145410 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.145380    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 29 10:37:15.145605 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.145411    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:37:15.145605 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.145454    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:37:15.145777 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.145466    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 29 10:37:15.149979 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.149940    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:15.153095 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.153068    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"]
Jul 29 10:37:15.226524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.226323    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.226524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.226358    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkm6\" (UniqueName: \"kubernetes.io/projected/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-api-access-nzkm6\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.226524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.226406    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.226524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.226422    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.226524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.226438    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.315951 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.315903    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"]
Jul 29 10:37:15.320856 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:37:15.320828    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb7ffb2_19af_4e74_b44b_1dbb70855951.slice/crio-f422c9a79c19fd74087eb3856759cd92c0af38bc9a3afc112245ae9df268b0bf WatchSource:0}: Error finding container f422c9a79c19fd74087eb3856759cd92c0af38bc9a3afc112245ae9df268b0bf: Status 404 returned error can't find the container with id f422c9a79c19fd74087eb3856759cd92c0af38bc9a3afc112245ae9df268b0bf
Jul 29 10:37:15.327441 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.327153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.327441 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.327202    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.327441 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.327233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.327441 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.327284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.327441 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.327312    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkm6\" (UniqueName: \"kubernetes.io/projected/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-api-access-nzkm6\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.327441 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:37:15.327407    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 29 10:37:15.327809 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:37:15.327498    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-proxy-tls podName:00c8b1c9-1c37-41f0-a7cf-c8e993ed1808 nodeName:}" failed. No retries permitted until 2026-07-29 10:37:15.827470316 +0000 UTC m=+518.055716848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-proxy-tls") pod "isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" (UID: "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 29 10:37:15.328327 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.328108    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.328426 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.328343    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.328762 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.328741    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.335098 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.335071    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkm6\" (UniqueName: \"kubernetes.io/projected/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-api-access-nzkm6\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.831030 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.830994    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:15.833468 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:15.833439    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-77b6fbc468-69w52\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:16.057553 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:16.057122    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:16.276493 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:16.276384    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"]
Jul 29 10:37:16.282622 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:16.282518    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" event={"ID":"fcb7ffb2-19af-4e74-b44b-1dbb70855951","Type":"ContainerStarted","Data":"f422c9a79c19fd74087eb3856759cd92c0af38bc9a3afc112245ae9df268b0bf"}
Jul 29 10:37:17.293791 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:17.293744    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" event={"ID":"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808","Type":"ContainerStarted","Data":"5517b1092bff038d8e3e744ab14d8369ad0f3e1b4ed63fe7376de448b9f51f53"}
Jul 29 10:37:29.350863 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:29.350763    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" event={"ID":"fcb7ffb2-19af-4e74-b44b-1dbb70855951","Type":"ContainerStarted","Data":"34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d"}
Jul 29 10:37:29.352329 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:29.352297    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" event={"ID":"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808","Type":"ContainerStarted","Data":"84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7"}
Jul 29 10:37:32.364630 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:32.364590    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" event={"ID":"fcb7ffb2-19af-4e74-b44b-1dbb70855951","Type":"ContainerStarted","Data":"8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237"}
Jul 29 10:37:32.365070 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:32.364655    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:32.365070 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:32.364722    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:32.366231 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:32.366201    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 29 10:37:32.383052 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:32.382996    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" podStartSLOduration=2.389287313 podStartE2EDuration="18.382978696s" totalImagesPullingTime="15.993691383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:32.381758843 +0000 UTC m=+534.610005392" watchObservedRunningTime="2026-07-29 10:37:32.382978696 +0000 UTC m=+534.611225246"
Jul 29 10:37:33.369072 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:33.369039    2572 generic.go:356] "Generic (PLEG): container finished" podID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerID="84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7" exitCode=0
Jul 29 10:37:33.369474 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:33.369119    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" event={"ID":"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808","Type":"ContainerDied","Data":"84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7"}
Jul 29 10:37:33.369601 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:33.369578    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 29 10:37:38.375714 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:38.375678    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:37:38.376287 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:38.376260    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 29 10:37:40.399865 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:40.399827    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" event={"ID":"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808","Type":"ContainerStarted","Data":"384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2"}
Jul 29 10:37:40.400370 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:40.399901    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" event={"ID":"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808","Type":"ContainerStarted","Data":"39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526"}
Jul 29 10:37:40.400370 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:40.400151    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:40.420449 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:40.420391    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podStartSLOduration=2.64850567 podStartE2EDuration="25.420372764s" totalImagesPullingTime="18.976286396s" totalInitContainerRuntime="3.795580698s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:40.418253629 +0000 UTC m=+542.646500178" watchObservedRunningTime="2026-07-29 10:37:40.420372764 +0000 UTC m=+542.648619315"
Jul 29 10:37:41.404169 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:41.404127    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:41.405427 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:41.405401    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:37:42.407448 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:42.407406    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:37:47.414421 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:47.414380    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:37:47.415084 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:47.415051    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:37:48.376389 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:48.376345    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 29 10:37:57.415749 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:57.415699    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:37:58.377067 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:37:58.377015    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 29 10:38:07.414894 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:07.414852    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:38:08.377092 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:08.377042    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.16:8080: connect: connection refused"
Jul 29 10:38:17.415592 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:17.415547    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:38:18.377698 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:18.377667    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:38:27.415773 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:27.415726    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:38:34.626973 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.626914    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"]
Jul 29 10:38:34.630082 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.630057    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:34.632187 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.632163    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f0182-kube-rbac-proxy-sar-config\""
Jul 29 10:38:34.632329 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.632188    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-f0182-serving-cert\""
Jul 29 10:38:34.638147 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.638123    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"]
Jul 29 10:38:34.770652 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.770601    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0d8268-5080-4e2e-9058-949bfba19c7a-proxy-tls\") pod \"switch-graph-f0182-68659b478c-prr6c\" (UID: \"bf0d8268-5080-4e2e-9058-949bfba19c7a\") " pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:34.770652 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.770659    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0d8268-5080-4e2e-9058-949bfba19c7a-openshift-service-ca-bundle\") pod \"switch-graph-f0182-68659b478c-prr6c\" (UID: \"bf0d8268-5080-4e2e-9058-949bfba19c7a\") " pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:34.871911 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.871871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0d8268-5080-4e2e-9058-949bfba19c7a-openshift-service-ca-bundle\") pod \"switch-graph-f0182-68659b478c-prr6c\" (UID: \"bf0d8268-5080-4e2e-9058-949bfba19c7a\") " pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:34.872051 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.872006    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0d8268-5080-4e2e-9058-949bfba19c7a-proxy-tls\") pod \"switch-graph-f0182-68659b478c-prr6c\" (UID: \"bf0d8268-5080-4e2e-9058-949bfba19c7a\") " pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:34.872683 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.872652    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0d8268-5080-4e2e-9058-949bfba19c7a-openshift-service-ca-bundle\") pod \"switch-graph-f0182-68659b478c-prr6c\" (UID: \"bf0d8268-5080-4e2e-9058-949bfba19c7a\") " pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:34.874422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.874402    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0d8268-5080-4e2e-9058-949bfba19c7a-proxy-tls\") pod \"switch-graph-f0182-68659b478c-prr6c\" (UID: \"bf0d8268-5080-4e2e-9058-949bfba19c7a\") " pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:34.941580 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:34.941488    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:35.079396 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:35.079196    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"]
Jul 29 10:38:35.082184 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:38:35.082153    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0d8268_5080_4e2e_9058_949bfba19c7a.slice/crio-62c208113cc299d2b29b433a0bdbfcd9cf0c0e0aeec6b153b97981b249757f02 WatchSource:0}: Error finding container 62c208113cc299d2b29b433a0bdbfcd9cf0c0e0aeec6b153b97981b249757f02: Status 404 returned error can't find the container with id 62c208113cc299d2b29b433a0bdbfcd9cf0c0e0aeec6b153b97981b249757f02
Jul 29 10:38:35.605812 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:35.605775    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" event={"ID":"bf0d8268-5080-4e2e-9058-949bfba19c7a","Type":"ContainerStarted","Data":"62c208113cc299d2b29b433a0bdbfcd9cf0c0e0aeec6b153b97981b249757f02"}
Jul 29 10:38:37.415910 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:37.415874    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:38:37.615250 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:37.615204    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" event={"ID":"bf0d8268-5080-4e2e-9058-949bfba19c7a","Type":"ContainerStarted","Data":"87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2"}
Jul 29 10:38:37.615437 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:37.615320    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:37.633161 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:37.633106    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" podStartSLOduration=1.320496693 podStartE2EDuration="3.633089812s" totalImagesPullingTime="2.312593119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:37.631771425 +0000 UTC m=+599.860017975" watchObservedRunningTime="2026-07-29 10:38:37.633089812 +0000 UTC m=+599.861336362"
Jul 29 10:38:43.624127 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:43.624092    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:38:44.794639 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:44.794604    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"]
Jul 29 10:38:44.795137 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:44.794828    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182" containerID="cri-o://87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2" gracePeriod=30
Jul 29 10:38:44.906239 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:44.906197    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"]
Jul 29 10:38:44.906515 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:44.906487    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container" containerID="cri-o://34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d" gracePeriod=30
Jul 29 10:38:44.906643 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:44.906591    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kube-rbac-proxy" containerID="cri-o://8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237" gracePeriod=30
Jul 29 10:38:44.998191 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:44.998150    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"]
Jul 29 10:38:45.002173 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.002144    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.004269 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.004240    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0a743-kube-rbac-proxy-sar-config\""
Jul 29 10:38:45.004414 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.004240    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0a743-predictor-serving-cert\""
Jul 29 10:38:45.010891 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.010860    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"]
Jul 29 10:38:45.164908 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.164871    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ccf183-bc4e-47ee-a353-a4696239b953-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0a743-predictor-98f985b96-6lt86\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") " pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.164908 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.164917    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ccf183-bc4e-47ee-a353-a4696239b953-proxy-tls\") pod \"success-200-isvc-0a743-predictor-98f985b96-6lt86\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") " pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.165220 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.164987    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrs8v\" (UniqueName: \"kubernetes.io/projected/58ccf183-bc4e-47ee-a353-a4696239b953-kube-api-access-vrs8v\") pod \"success-200-isvc-0a743-predictor-98f985b96-6lt86\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") " pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.266149 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.266106    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ccf183-bc4e-47ee-a353-a4696239b953-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0a743-predictor-98f985b96-6lt86\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") " pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.266149 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.266156    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ccf183-bc4e-47ee-a353-a4696239b953-proxy-tls\") pod \"success-200-isvc-0a743-predictor-98f985b96-6lt86\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") " pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.266400 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.266185    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrs8v\" (UniqueName: \"kubernetes.io/projected/58ccf183-bc4e-47ee-a353-a4696239b953-kube-api-access-vrs8v\") pod \"success-200-isvc-0a743-predictor-98f985b96-6lt86\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") " pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.266853 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.266825    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ccf183-bc4e-47ee-a353-a4696239b953-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0a743-predictor-98f985b96-6lt86\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") " pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.268751 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.268722    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ccf183-bc4e-47ee-a353-a4696239b953-proxy-tls\") pod \"success-200-isvc-0a743-predictor-98f985b96-6lt86\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") " pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.273796 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.273768    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrs8v\" (UniqueName: \"kubernetes.io/projected/58ccf183-bc4e-47ee-a353-a4696239b953-kube-api-access-vrs8v\") pod \"success-200-isvc-0a743-predictor-98f985b96-6lt86\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") " pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.314358 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.314323    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.457204 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.457107    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"]
Jul 29 10:38:45.461043 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:38:45.461014    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ccf183_bc4e_47ee_a353_a4696239b953.slice/crio-1db0a831776fb8cca798b6a7f934e10b1bbeee2d33b7e9de7433ca6775d0fdf1 WatchSource:0}: Error finding container 1db0a831776fb8cca798b6a7f934e10b1bbeee2d33b7e9de7433ca6775d0fdf1: Status 404 returned error can't find the container with id 1db0a831776fb8cca798b6a7f934e10b1bbeee2d33b7e9de7433ca6775d0fdf1
Jul 29 10:38:45.649249 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.649217    2572 generic.go:356] "Generic (PLEG): container finished" podID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerID="8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237" exitCode=2
Jul 29 10:38:45.649420 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.649288    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" event={"ID":"fcb7ffb2-19af-4e74-b44b-1dbb70855951","Type":"ContainerDied","Data":"8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237"}
Jul 29 10:38:45.650990 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.650967    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" event={"ID":"58ccf183-bc4e-47ee-a353-a4696239b953","Type":"ContainerStarted","Data":"0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602"}
Jul 29 10:38:45.651105 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.651020    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" event={"ID":"58ccf183-bc4e-47ee-a353-a4696239b953","Type":"ContainerStarted","Data":"518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7"}
Jul 29 10:38:45.651105 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.651051    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" event={"ID":"58ccf183-bc4e-47ee-a353-a4696239b953","Type":"ContainerStarted","Data":"1db0a831776fb8cca798b6a7f934e10b1bbeee2d33b7e9de7433ca6775d0fdf1"}
Jul 29 10:38:45.651105 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.651088    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:45.668714 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:45.668654    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podStartSLOduration=1.668638519 podStartE2EDuration="1.668638519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:45.666797824 +0000 UTC m=+607.895044375" watchObservedRunningTime="2026-07-29 10:38:45.668638519 +0000 UTC m=+607.896885069"
Jul 29 10:38:46.654870 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:46.654834    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:46.656355 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:46.656324    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 29 10:38:47.415511 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:47.415460    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:38:47.658881 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:47.658839    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 29 10:38:48.268873 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.268844    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:38:48.395445 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.395415    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-rbac-proxy-sar-config\") pod \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") "
Jul 29 10:38:48.395643 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.395487    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb7ffb2-19af-4e74-b44b-1dbb70855951-proxy-tls\") pod \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") "
Jul 29 10:38:48.395643 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.395538    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn6gf\" (UniqueName: \"kubernetes.io/projected/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-api-access-rn6gf\") pod \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\" (UID: \"fcb7ffb2-19af-4e74-b44b-1dbb70855951\") "
Jul 29 10:38:48.395824 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.395799    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcb7ffb2-19af-4e74-b44b-1dbb70855951" (UID: "fcb7ffb2-19af-4e74-b44b-1dbb70855951"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:38:48.397765 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.397739    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb7ffb2-19af-4e74-b44b-1dbb70855951-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcb7ffb2-19af-4e74-b44b-1dbb70855951" (UID: "fcb7ffb2-19af-4e74-b44b-1dbb70855951"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:38:48.397867 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.397825    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-api-access-rn6gf" (OuterVolumeSpecName: "kube-api-access-rn6gf") pod "fcb7ffb2-19af-4e74-b44b-1dbb70855951" (UID: "fcb7ffb2-19af-4e74-b44b-1dbb70855951"). InnerVolumeSpecName "kube-api-access-rn6gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:38:48.496272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.496228    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn6gf\" (UniqueName: \"kubernetes.io/projected/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-api-access-rn6gf\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:38:48.496272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.496264    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcb7ffb2-19af-4e74-b44b-1dbb70855951-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:38:48.496272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.496277    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcb7ffb2-19af-4e74-b44b-1dbb70855951-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:38:48.622784 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.622741    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:38:48.663725 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.663635    2572 generic.go:356] "Generic (PLEG): container finished" podID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerID="34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d" exitCode=0
Jul 29 10:38:48.663725 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.663694    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"
Jul 29 10:38:48.663725 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.663690    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" event={"ID":"fcb7ffb2-19af-4e74-b44b-1dbb70855951","Type":"ContainerDied","Data":"34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d"}
Jul 29 10:38:48.664236 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.663759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx" event={"ID":"fcb7ffb2-19af-4e74-b44b-1dbb70855951","Type":"ContainerDied","Data":"f422c9a79c19fd74087eb3856759cd92c0af38bc9a3afc112245ae9df268b0bf"}
Jul 29 10:38:48.664236 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.663788    2572 scope.go:117] "RemoveContainer" containerID="8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237"
Jul 29 10:38:48.672484 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.672464    2572 scope.go:117] "RemoveContainer" containerID="34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d"
Jul 29 10:38:48.680370 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.680352    2572 scope.go:117] "RemoveContainer" containerID="8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237"
Jul 29 10:38:48.680672 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:38:48.680648    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237\": container with ID starting with 8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237 not found: ID does not exist" containerID="8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237"
Jul 29 10:38:48.680737 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.680682    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237"} err="failed to get container status \"8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237\": rpc error: code = NotFound desc = could not find container \"8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237\": container with ID starting with 8acdb3d4ce60651a3e86716107160987b63e28b6e89c9750cf8e99d95a027237 not found: ID does not exist"
Jul 29 10:38:48.680737 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.680719    2572 scope.go:117] "RemoveContainer" containerID="34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d"
Jul 29 10:38:48.681020 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:38:48.680992    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d\": container with ID starting with 34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d not found: ID does not exist" containerID="34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d"
Jul 29 10:38:48.681110 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.681025    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d"} err="failed to get container status \"34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d\": rpc error: code = NotFound desc = could not find container \"34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d\": container with ID starting with 34f8dcebdecc20cd291745f3067316a3f53b87f3080a1b31539cf3288bbc256d not found: ID does not exist"
Jul 29 10:38:48.684071 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.684046    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"]
Jul 29 10:38:48.685860 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:48.685837    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0182-predictor-54bbdb7d5b-vj5fx"]
Jul 29 10:38:50.310084 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:50.310048    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" path="/var/lib/kubelet/pods/fcb7ffb2-19af-4e74-b44b-1dbb70855951/volumes"
Jul 29 10:38:51.305699 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:51.305663    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 29 10:38:52.665228 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:52.665196    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:38:52.665691 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:52.665666    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 29 10:38:53.623206 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:53.623163    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:38:58.623312 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:58.623270    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:38:58.623759 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:38:58.623370    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:39:01.306188 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:01.306100    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:39:02.666324 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:02.666276    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 29 10:39:03.623133 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:03.623094    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:39:08.622937 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:08.622883    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:39:12.666584 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:12.666532    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 29 10:39:13.622985 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:13.622946    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:39:14.952372 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:14.952345    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:39:14.993156 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:14.993110    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0d8268-5080-4e2e-9058-949bfba19c7a-proxy-tls\") pod \"bf0d8268-5080-4e2e-9058-949bfba19c7a\" (UID: \"bf0d8268-5080-4e2e-9058-949bfba19c7a\") "
Jul 29 10:39:14.993351 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:14.993195    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0d8268-5080-4e2e-9058-949bfba19c7a-openshift-service-ca-bundle\") pod \"bf0d8268-5080-4e2e-9058-949bfba19c7a\" (UID: \"bf0d8268-5080-4e2e-9058-949bfba19c7a\") "
Jul 29 10:39:14.993556 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:14.993521    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0d8268-5080-4e2e-9058-949bfba19c7a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bf0d8268-5080-4e2e-9058-949bfba19c7a" (UID: "bf0d8268-5080-4e2e-9058-949bfba19c7a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:39:14.995285 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:14.995257    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf0d8268-5080-4e2e-9058-949bfba19c7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf0d8268-5080-4e2e-9058-949bfba19c7a" (UID: "bf0d8268-5080-4e2e-9058-949bfba19c7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:39:15.094455 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.094353    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf0d8268-5080-4e2e-9058-949bfba19c7a-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:39:15.094455 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.094387    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf0d8268-5080-4e2e-9058-949bfba19c7a-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:39:15.765014 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.764969    2572 generic.go:356] "Generic (PLEG): container finished" podID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerID="87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2" exitCode=0
Jul 29 10:39:15.765223 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.765022    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"
Jul 29 10:39:15.765223 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.765055    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" event={"ID":"bf0d8268-5080-4e2e-9058-949bfba19c7a","Type":"ContainerDied","Data":"87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2"}
Jul 29 10:39:15.765223 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.765126    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c" event={"ID":"bf0d8268-5080-4e2e-9058-949bfba19c7a","Type":"ContainerDied","Data":"62c208113cc299d2b29b433a0bdbfcd9cf0c0e0aeec6b153b97981b249757f02"}
Jul 29 10:39:15.765223 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.765157    2572 scope.go:117] "RemoveContainer" containerID="87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2"
Jul 29 10:39:15.773437 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.773413    2572 scope.go:117] "RemoveContainer" containerID="87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2"
Jul 29 10:39:15.773737 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:15.773716    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2\": container with ID starting with 87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2 not found: ID does not exist" containerID="87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2"
Jul 29 10:39:15.773826 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.773747    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2"} err="failed to get container status \"87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2\": rpc error: code = NotFound desc = could not find container \"87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2\": container with ID starting with 87b91b7454e6a01224370468df557e25efac4be29c34b1f5e699ad0c58d237e2 not found: ID does not exist"
Jul 29 10:39:15.785328 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.785292    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"]
Jul 29 10:39:15.789171 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:15.789143    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-f0182-68659b478c-prr6c"]
Jul 29 10:39:16.310094 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:16.310054    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" path="/var/lib/kubelet/pods/bf0d8268-5080-4e2e-9058-949bfba19c7a/volumes"
Jul 29 10:39:22.666217 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:22.666172    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 29 10:39:32.666552 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:32.666517    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:39:34.613876 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.613839    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"]
Jul 29 10:39:34.614271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.614124    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182"
Jul 29 10:39:34.614271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.614136    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182"
Jul 29 10:39:34.614271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.614147    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kube-rbac-proxy"
Jul 29 10:39:34.614271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.614152    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kube-rbac-proxy"
Jul 29 10:39:34.614271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.614165    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container"
Jul 29 10:39:34.614271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.614171    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container"
Jul 29 10:39:34.614271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.614214    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kserve-container"
Jul 29 10:39:34.614271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.614222    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb7ffb2-19af-4e74-b44b-1dbb70855951" containerName="kube-rbac-proxy"
Jul 29 10:39:34.614271 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.614230    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0d8268-5080-4e2e-9058-949bfba19c7a" containerName="switch-graph-f0182"
Jul 29 10:39:34.617102 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.617075    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:34.619194 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.619171    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 29 10:39:34.619299 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.619221    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 29 10:39:34.626612 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.626581    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"]
Jul 29 10:39:34.635368 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.635338    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335eb5f3-3f74-4c50-b93e-0eca3d60899e-proxy-tls\") pod \"model-chainer-79dd7c6579-pzmm2\" (UID: \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\") " pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:34.635533 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.635392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/335eb5f3-3f74-4c50-b93e-0eca3d60899e-openshift-service-ca-bundle\") pod \"model-chainer-79dd7c6579-pzmm2\" (UID: \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\") " pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:34.735766 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.735723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/335eb5f3-3f74-4c50-b93e-0eca3d60899e-openshift-service-ca-bundle\") pod \"model-chainer-79dd7c6579-pzmm2\" (UID: \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\") " pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:34.735989 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.735794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335eb5f3-3f74-4c50-b93e-0eca3d60899e-proxy-tls\") pod \"model-chainer-79dd7c6579-pzmm2\" (UID: \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\") " pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:34.736379 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.736358    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/335eb5f3-3f74-4c50-b93e-0eca3d60899e-openshift-service-ca-bundle\") pod \"model-chainer-79dd7c6579-pzmm2\" (UID: \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\") " pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:34.738230 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.738212    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335eb5f3-3f74-4c50-b93e-0eca3d60899e-proxy-tls\") pod \"model-chainer-79dd7c6579-pzmm2\" (UID: \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\") " pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:34.928630 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:34.928590    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:35.060087 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:35.060053    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"]
Jul 29 10:39:35.063366 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:39:35.063335    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335eb5f3_3f74_4c50_b93e_0eca3d60899e.slice/crio-cabf764974f9c651c8d6a9a1ba1cbe0df8bcf890ff66f63847907efff9f44c97 WatchSource:0}: Error finding container cabf764974f9c651c8d6a9a1ba1cbe0df8bcf890ff66f63847907efff9f44c97: Status 404 returned error can't find the container with id cabf764974f9c651c8d6a9a1ba1cbe0df8bcf890ff66f63847907efff9f44c97
Jul 29 10:39:35.840746 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:35.840713    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" event={"ID":"335eb5f3-3f74-4c50-b93e-0eca3d60899e","Type":"ContainerStarted","Data":"c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00"}
Jul 29 10:39:35.841227 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:35.840762    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" event={"ID":"335eb5f3-3f74-4c50-b93e-0eca3d60899e","Type":"ContainerStarted","Data":"cabf764974f9c651c8d6a9a1ba1cbe0df8bcf890ff66f63847907efff9f44c97"}
Jul 29 10:39:35.841227 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:35.840807    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:35.856513 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:35.856456    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" podStartSLOduration=1.856436654 podStartE2EDuration="1.856436654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:35.855138435 +0000 UTC m=+658.083384984" watchObservedRunningTime="2026-07-29 10:39:35.856436654 +0000 UTC m=+658.084683205"
Jul 29 10:39:41.850589 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:41.850557    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:39:44.702311 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.702273    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"]
Jul 29 10:39:44.702713 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.702474    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer" containerID="cri-o://c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00" gracePeriod=30
Jul 29 10:39:44.832998 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.832951    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"]
Jul 29 10:39:44.833330 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.833261    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container" containerID="cri-o://39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526" gracePeriod=30
Jul 29 10:39:44.833468 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.833291    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kube-rbac-proxy" containerID="cri-o://384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2" gracePeriod=30
Jul 29 10:39:44.893655 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.893618    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"]
Jul 29 10:39:44.898684 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.898656    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:44.900813 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.900780    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f1d40-predictor-serving-cert\""
Jul 29 10:39:44.900813 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.900802    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f1d40-kube-rbac-proxy-sar-config\""
Jul 29 10:39:44.907951 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:44.907906    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"]
Jul 29 10:39:45.022355 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.022265    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:45.022355 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.022333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:45.022569 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.022363    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-897sq\" (UniqueName: \"kubernetes.io/projected/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-api-access-897sq\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:45.074232 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.074190    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"]
Jul 29 10:39:45.077772 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.077746    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:45.079747 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.079720    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-0a743-serving-cert\""
Jul 29 10:39:45.079747 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.079742    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-0a743-kube-rbac-proxy-sar-config\""
Jul 29 10:39:45.087304 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.087215    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"]
Jul 29 10:39:45.122978 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.122939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d991797-5a7c-439d-af61-5d7f08d8b15a-proxy-tls\") pod \"switch-graph-0a743-9d56587d4-sqlmx\" (UID: \"4d991797-5a7c-439d-af61-5d7f08d8b15a\") " pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:45.122978 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.122993    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:45.123255 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.123057    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:45.123255 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.123096    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-897sq\" (UniqueName: \"kubernetes.io/projected/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-api-access-897sq\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:45.123255 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.123148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d991797-5a7c-439d-af61-5d7f08d8b15a-openshift-service-ca-bundle\") pod \"switch-graph-0a743-9d56587d4-sqlmx\" (UID: \"4d991797-5a7c-439d-af61-5d7f08d8b15a\") " pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:45.123255 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:45.123170    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-serving-cert: secret "success-200-isvc-f1d40-predictor-serving-cert" not found
Jul 29 10:39:45.123255 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:45.123247    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls podName:d7ca144f-8def-4bac-a45f-89d3695e9928 nodeName:}" failed. No retries permitted until 2026-07-29 10:39:45.623224999 +0000 UTC m=+667.851471540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls") pod "success-200-isvc-f1d40-predictor-58f666b764-89zbs" (UID: "d7ca144f-8def-4bac-a45f-89d3695e9928") : secret "success-200-isvc-f1d40-predictor-serving-cert" not found
Jul 29 10:39:45.123759 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.123739    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:45.131470 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.131447    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-897sq\" (UniqueName: \"kubernetes.io/projected/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-api-access-897sq\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:45.223888 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.223837    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d991797-5a7c-439d-af61-5d7f08d8b15a-proxy-tls\") pod \"switch-graph-0a743-9d56587d4-sqlmx\" (UID: \"4d991797-5a7c-439d-af61-5d7f08d8b15a\") " pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:45.224083 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.224004    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d991797-5a7c-439d-af61-5d7f08d8b15a-openshift-service-ca-bundle\") pod \"switch-graph-0a743-9d56587d4-sqlmx\" (UID: \"4d991797-5a7c-439d-af61-5d7f08d8b15a\") " pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:45.224083 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:45.224034    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-0a743-serving-cert: secret "switch-graph-0a743-serving-cert" not found
Jul 29 10:39:45.224164 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:45.224132    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d991797-5a7c-439d-af61-5d7f08d8b15a-proxy-tls podName:4d991797-5a7c-439d-af61-5d7f08d8b15a nodeName:}" failed. No retries permitted until 2026-07-29 10:39:45.724109713 +0000 UTC m=+667.952356244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d991797-5a7c-439d-af61-5d7f08d8b15a-proxy-tls") pod "switch-graph-0a743-9d56587d4-sqlmx" (UID: "4d991797-5a7c-439d-af61-5d7f08d8b15a") : secret "switch-graph-0a743-serving-cert" not found
Jul 29 10:39:45.224686 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.224664    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d991797-5a7c-439d-af61-5d7f08d8b15a-openshift-service-ca-bundle\") pod \"switch-graph-0a743-9d56587d4-sqlmx\" (UID: \"4d991797-5a7c-439d-af61-5d7f08d8b15a\") " pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:45.628070 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.628022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:45.628265 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:45.628148    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-serving-cert: secret "success-200-isvc-f1d40-predictor-serving-cert" not found
Jul 29 10:39:45.628265 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:45.628211    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls podName:d7ca144f-8def-4bac-a45f-89d3695e9928 nodeName:}" failed. No retries permitted until 2026-07-29 10:39:46.628197352 +0000 UTC m=+668.856443880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls") pod "success-200-isvc-f1d40-predictor-58f666b764-89zbs" (UID: "d7ca144f-8def-4bac-a45f-89d3695e9928") : secret "success-200-isvc-f1d40-predictor-serving-cert" not found
Jul 29 10:39:45.728969 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.728904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d991797-5a7c-439d-af61-5d7f08d8b15a-proxy-tls\") pod \"switch-graph-0a743-9d56587d4-sqlmx\" (UID: \"4d991797-5a7c-439d-af61-5d7f08d8b15a\") " pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:45.731414 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.731391    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d991797-5a7c-439d-af61-5d7f08d8b15a-proxy-tls\") pod \"switch-graph-0a743-9d56587d4-sqlmx\" (UID: \"4d991797-5a7c-439d-af61-5d7f08d8b15a\") " pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:45.880918 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.880829    2572 generic.go:356] "Generic (PLEG): container finished" podID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerID="384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2" exitCode=2
Jul 29 10:39:45.880918 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.880895    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" event={"ID":"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808","Type":"ContainerDied","Data":"384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2"}
Jul 29 10:39:45.990412 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:45.990354    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:46.142714 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.142573    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"]
Jul 29 10:39:46.145353 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:39:46.145328    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d991797_5a7c_439d_af61_5d7f08d8b15a.slice/crio-a40e2c160edf2b54821c00436ac46576e62df638b8b24d7c254e64f97eddc5a6 WatchSource:0}: Error finding container a40e2c160edf2b54821c00436ac46576e62df638b8b24d7c254e64f97eddc5a6: Status 404 returned error can't find the container with id a40e2c160edf2b54821c00436ac46576e62df638b8b24d7c254e64f97eddc5a6
Jul 29 10:39:46.636110 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.636065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:46.638586 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.638566    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls\") pod \"success-200-isvc-f1d40-predictor-58f666b764-89zbs\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") " pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:46.710034 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.709999    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:46.848364 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.848323    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:39:46.861883 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.861848    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"]
Jul 29 10:39:46.864917 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:39:46.864868    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ca144f_8def_4bac_a45f_89d3695e9928.slice/crio-26b8013f2b7aac73b06b9b2682fcffa793bc1c3b2c86be05735dc9b11ca7318f WatchSource:0}: Error finding container 26b8013f2b7aac73b06b9b2682fcffa793bc1c3b2c86be05735dc9b11ca7318f: Status 404 returned error can't find the container with id 26b8013f2b7aac73b06b9b2682fcffa793bc1c3b2c86be05735dc9b11ca7318f
Jul 29 10:39:46.886905 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.886825    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" event={"ID":"4d991797-5a7c-439d-af61-5d7f08d8b15a","Type":"ContainerStarted","Data":"383a7c3fda1b23311b81859262d55f529f090ed1a46229a3e43acbd0c0b1303d"}
Jul 29 10:39:46.887057 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.886910    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:46.887057 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.886955    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" event={"ID":"4d991797-5a7c-439d-af61-5d7f08d8b15a","Type":"ContainerStarted","Data":"a40e2c160edf2b54821c00436ac46576e62df638b8b24d7c254e64f97eddc5a6"}
Jul 29 10:39:46.888051 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.888030    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" event={"ID":"d7ca144f-8def-4bac-a45f-89d3695e9928","Type":"ContainerStarted","Data":"26b8013f2b7aac73b06b9b2682fcffa793bc1c3b2c86be05735dc9b11ca7318f"}
Jul 29 10:39:46.902513 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:46.902464    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" podStartSLOduration=1.902445344 podStartE2EDuration="1.902445344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:46.900726395 +0000 UTC m=+669.128972960" watchObservedRunningTime="2026-07-29 10:39:46.902445344 +0000 UTC m=+669.130691893"
Jul 29 10:39:47.407969 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:47.407902    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Jul 29 10:39:47.893937 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:47.893880    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" event={"ID":"d7ca144f-8def-4bac-a45f-89d3695e9928","Type":"ContainerStarted","Data":"b4b964c0a9b09558ca682ef10e27b3d31267e3e22d10123d551e5cfe576581c0"}
Jul 29 10:39:47.894365 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:47.893964    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" event={"ID":"d7ca144f-8def-4bac-a45f-89d3695e9928","Type":"ContainerStarted","Data":"6f8d32337ad6507135892570ef3608a5fc97f5cc20be1b97f4c293af473bf2f9"}
Jul 29 10:39:47.894365 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:47.894237    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:47.894365 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:47.894294    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:47.895422 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:47.895394    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 29 10:39:47.914568 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:47.914501    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" podStartSLOduration=3.914475989 podStartE2EDuration="3.914475989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:47.914049753 +0000 UTC m=+670.142296327" watchObservedRunningTime="2026-07-29 10:39:47.914475989 +0000 UTC m=+670.142722541"
Jul 29 10:39:48.897309 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:48.897270    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 29 10:39:49.891688 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.891659    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:39:49.902447 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.902410    2572 generic.go:356] "Generic (PLEG): container finished" podID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerID="39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526" exitCode=0
Jul 29 10:39:49.902823 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.902480    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"
Jul 29 10:39:49.902823 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.902490    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" event={"ID":"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808","Type":"ContainerDied","Data":"39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526"}
Jul 29 10:39:49.902823 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.902558    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52" event={"ID":"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808","Type":"ContainerDied","Data":"5517b1092bff038d8e3e744ab14d8369ad0f3e1b4ed63fe7376de448b9f51f53"}
Jul 29 10:39:49.902823 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.902586    2572 scope.go:117] "RemoveContainer" containerID="384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2"
Jul 29 10:39:49.910303 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.910281    2572 scope.go:117] "RemoveContainer" containerID="39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526"
Jul 29 10:39:49.920374 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.920345    2572 scope.go:117] "RemoveContainer" containerID="84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7"
Jul 29 10:39:49.930557 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.930534    2572 scope.go:117] "RemoveContainer" containerID="384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2"
Jul 29 10:39:49.930899 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:49.930877    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2\": container with ID starting with 384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2 not found: ID does not exist" containerID="384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2"
Jul 29 10:39:49.931008 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.930908    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2"} err="failed to get container status \"384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2\": rpc error: code = NotFound desc = could not find container \"384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2\": container with ID starting with 384cecdc664ea111c92f3dc9c35c47f3dac44915b348a80ce4e7951d54b72aa2 not found: ID does not exist"
Jul 29 10:39:49.931008 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.930967    2572 scope.go:117] "RemoveContainer" containerID="39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526"
Jul 29 10:39:49.931279 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:49.931258    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526\": container with ID starting with 39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526 not found: ID does not exist" containerID="39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526"
Jul 29 10:39:49.931352 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.931290    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526"} err="failed to get container status \"39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526\": rpc error: code = NotFound desc = could not find container \"39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526\": container with ID starting with 39348facc94b2087cb43967fbb5c83546310b9269e6b7f121607acd1ac738526 not found: ID does not exist"
Jul 29 10:39:49.931352 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.931314    2572 scope.go:117] "RemoveContainer" containerID="84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7"
Jul 29 10:39:49.931589 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:39:49.931571    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7\": container with ID starting with 84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7 not found: ID does not exist" containerID="84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7"
Jul 29 10:39:49.931650 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.931595    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7"} err="failed to get container status \"84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7\": rpc error: code = NotFound desc = could not find container \"84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7\": container with ID starting with 84e03a57ec30762b026cb5af29c0f976960b8c8d4af8cf0ba132a0bb2e81bcc7 not found: ID does not exist"
Jul 29 10:39:49.963910 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.963877    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kserve-provision-location\") pod \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") "
Jul 29 10:39:49.964094 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.963975    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-proxy-tls\") pod \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") "
Jul 29 10:39:49.964094 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.964002    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-rbac-proxy-sar-config\") pod \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") "
Jul 29 10:39:49.964094 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.964020    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-cabundle-cert\") pod \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") "
Jul 29 10:39:49.964229 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.964191    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzkm6\" (UniqueName: \"kubernetes.io/projected/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-api-access-nzkm6\") pod \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\" (UID: \"00c8b1c9-1c37-41f0-a7cf-c8e993ed1808\") "
Jul 29 10:39:49.964229 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.964216    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" (UID: "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:39:49.964358 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.964340    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" (UID: "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:39:49.964413 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.964377    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" (UID: "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:39:49.964462 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.964437    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:39:49.964462 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.964449    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-cabundle-cert\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:39:49.964462 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.964459    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kserve-provision-location\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:39:49.966146 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.966090    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" (UID: "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:39:49.966267 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:49.966247    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-api-access-nzkm6" (OuterVolumeSpecName: "kube-api-access-nzkm6") pod "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" (UID: "00c8b1c9-1c37-41f0-a7cf-c8e993ed1808"). InnerVolumeSpecName "kube-api-access-nzkm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:39:50.065816 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:50.065774    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:39:50.065816 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:50.065809    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzkm6\" (UniqueName: \"kubernetes.io/projected/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808-kube-api-access-nzkm6\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:39:50.224186 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:50.224091    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"]
Jul 29 10:39:50.230002 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:50.229966    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-77b6fbc468-69w52"]
Jul 29 10:39:50.310322 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:50.310285    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" path="/var/lib/kubelet/pods/00c8b1c9-1c37-41f0-a7cf-c8e993ed1808/volumes"
Jul 29 10:39:51.848737 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:51.848694    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:39:52.899808 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:52.899780    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:39:53.902273 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:53.902243    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:39:53.902912 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:53.902883    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 29 10:39:56.848518 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:56.848471    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:39:56.849055 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:39:56.848616    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:40:01.848674 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:01.848627    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:03.903269 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:03.903224    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 29 10:40:06.849079 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:06.849027    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:11.848157 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:11.848116    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:13.903744 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:13.903699    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 29 10:40:14.854480 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:14.854451    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:40:14.969595 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:14.969507    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/335eb5f3-3f74-4c50-b93e-0eca3d60899e-openshift-service-ca-bundle\") pod \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\" (UID: \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\") "
Jul 29 10:40:14.969595 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:14.969555    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335eb5f3-3f74-4c50-b93e-0eca3d60899e-proxy-tls\") pod \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\" (UID: \"335eb5f3-3f74-4c50-b93e-0eca3d60899e\") "
Jul 29 10:40:14.970005 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:14.969889    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335eb5f3-3f74-4c50-b93e-0eca3d60899e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "335eb5f3-3f74-4c50-b93e-0eca3d60899e" (UID: "335eb5f3-3f74-4c50-b93e-0eca3d60899e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:40:14.971598 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:14.971573    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335eb5f3-3f74-4c50-b93e-0eca3d60899e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "335eb5f3-3f74-4c50-b93e-0eca3d60899e" (UID: "335eb5f3-3f74-4c50-b93e-0eca3d60899e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:40:14.999911 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:14.999870    2572 generic.go:356] "Generic (PLEG): container finished" podID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerID="c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00" exitCode=0
Jul 29 10:40:14.999911 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:14.999912    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"
Jul 29 10:40:15.000180 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:14.999953    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" event={"ID":"335eb5f3-3f74-4c50-b93e-0eca3d60899e","Type":"ContainerDied","Data":"c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00"}
Jul 29 10:40:15.000180 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:15.000022    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2" event={"ID":"335eb5f3-3f74-4c50-b93e-0eca3d60899e","Type":"ContainerDied","Data":"cabf764974f9c651c8d6a9a1ba1cbe0df8bcf890ff66f63847907efff9f44c97"}
Jul 29 10:40:15.000180 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:15.000054    2572 scope.go:117] "RemoveContainer" containerID="c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00"
Jul 29 10:40:15.008793 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:15.008766    2572 scope.go:117] "RemoveContainer" containerID="c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00"
Jul 29 10:40:15.009102 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:40:15.009079    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00\": container with ID starting with c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00 not found: ID does not exist" containerID="c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00"
Jul 29 10:40:15.009167 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:15.009111    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00"} err="failed to get container status \"c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00\": rpc error: code = NotFound desc = could not find container \"c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00\": container with ID starting with c130d6345d4e35cf00f09eafb303aa2ef319406f1c452bd1291d9b65af33ae00 not found: ID does not exist"
Jul 29 10:40:15.019745 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:15.019724    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"]
Jul 29 10:40:15.022390 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:15.022368    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-79dd7c6579-pzmm2"]
Jul 29 10:40:15.071056 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:15.071018    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/335eb5f3-3f74-4c50-b93e-0eca3d60899e-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:40:15.071056 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:15.071050    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335eb5f3-3f74-4c50-b93e-0eca3d60899e-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:40:16.310315 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:16.310277    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" path="/var/lib/kubelet/pods/335eb5f3-3f74-4c50-b93e-0eca3d60899e/volumes"
Jul 29 10:40:23.903342 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:23.903302    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 29 10:40:33.904130 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:33.904046    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:40:44.894678 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.894641    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"]
Jul 29 10:40:44.895187 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895120    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kube-rbac-proxy"
Jul 29 10:40:44.895187 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895140    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kube-rbac-proxy"
Jul 29 10:40:44.895187 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895158    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container"
Jul 29 10:40:44.895187 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895167    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container"
Jul 29 10:40:44.895187 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895182    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer"
Jul 29 10:40:44.895187 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895191    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer"
Jul 29 10:40:44.895487 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895203    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="storage-initializer"
Jul 29 10:40:44.895487 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895212    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="storage-initializer"
Jul 29 10:40:44.895487 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895291    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kserve-container"
Jul 29 10:40:44.895487 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895304    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="335eb5f3-3f74-4c50-b93e-0eca3d60899e" containerName="model-chainer"
Jul 29 10:40:44.895487 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.895314    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c8b1c9-1c37-41f0-a7cf-c8e993ed1808" containerName="kube-rbac-proxy"
Jul 29 10:40:44.899414 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.899391    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:40:44.901297 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.901277    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f1d40-serving-cert\""
Jul 29 10:40:44.901428 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.901277    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f1d40-kube-rbac-proxy-sar-config\""
Jul 29 10:40:44.906142 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.906116    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"]
Jul 29 10:40:44.997010 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.996973    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772851ff-213c-4341-9eeb-85bcbe08cf6c-proxy-tls\") pod \"sequence-graph-f1d40-78bdff7fd9-5sdnw\" (UID: \"772851ff-213c-4341-9eeb-85bcbe08cf6c\") " pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:40:44.997186 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:44.997060    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/772851ff-213c-4341-9eeb-85bcbe08cf6c-openshift-service-ca-bundle\") pod \"sequence-graph-f1d40-78bdff7fd9-5sdnw\" (UID: \"772851ff-213c-4341-9eeb-85bcbe08cf6c\") " pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:40:45.098411 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:45.098366    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772851ff-213c-4341-9eeb-85bcbe08cf6c-proxy-tls\") pod \"sequence-graph-f1d40-78bdff7fd9-5sdnw\" (UID: \"772851ff-213c-4341-9eeb-85bcbe08cf6c\") " pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:40:45.098603 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:45.098457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/772851ff-213c-4341-9eeb-85bcbe08cf6c-openshift-service-ca-bundle\") pod \"sequence-graph-f1d40-78bdff7fd9-5sdnw\" (UID: \"772851ff-213c-4341-9eeb-85bcbe08cf6c\") " pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:40:45.099112 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:45.099087    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/772851ff-213c-4341-9eeb-85bcbe08cf6c-openshift-service-ca-bundle\") pod \"sequence-graph-f1d40-78bdff7fd9-5sdnw\" (UID: \"772851ff-213c-4341-9eeb-85bcbe08cf6c\") " pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:40:45.100942 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:45.100893    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772851ff-213c-4341-9eeb-85bcbe08cf6c-proxy-tls\") pod \"sequence-graph-f1d40-78bdff7fd9-5sdnw\" (UID: \"772851ff-213c-4341-9eeb-85bcbe08cf6c\") " pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:40:45.212149 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:45.212052    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:40:45.345743 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:45.345708    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"]
Jul 29 10:40:45.349087 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:40:45.349054    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772851ff_213c_4341_9eeb_85bcbe08cf6c.slice/crio-f948df69ce71f7951f985fa97d2f0eb1ccdb5baf1bffa1025d872077ddf66d2a WatchSource:0}: Error finding container f948df69ce71f7951f985fa97d2f0eb1ccdb5baf1bffa1025d872077ddf66d2a: Status 404 returned error can't find the container with id f948df69ce71f7951f985fa97d2f0eb1ccdb5baf1bffa1025d872077ddf66d2a
Jul 29 10:40:46.114544 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:46.114506    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" event={"ID":"772851ff-213c-4341-9eeb-85bcbe08cf6c","Type":"ContainerStarted","Data":"fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247"}
Jul 29 10:40:46.114955 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:46.114565    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" event={"ID":"772851ff-213c-4341-9eeb-85bcbe08cf6c","Type":"ContainerStarted","Data":"f948df69ce71f7951f985fa97d2f0eb1ccdb5baf1bffa1025d872077ddf66d2a"}
Jul 29 10:40:46.114955 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:46.114605    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:40:46.129649 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:46.129595    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" podStartSLOduration=2.129577135 podStartE2EDuration="2.129577135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:40:46.129083111 +0000 UTC m=+728.357329665" watchObservedRunningTime="2026-07-29 10:40:46.129577135 +0000 UTC m=+728.357823698"
Jul 29 10:40:52.123486 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:40:52.123452    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:47:59.771711 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.771671    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"]
Jul 29 10:47:59.772329 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.771887    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743" containerID="cri-o://383a7c3fda1b23311b81859262d55f529f090ed1a46229a3e43acbd0c0b1303d" gracePeriod=30
Jul 29 10:47:59.927287 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.927254    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"]
Jul 29 10:47:59.927554 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.927524    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container" containerID="cri-o://518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7" gracePeriod=30
Jul 29 10:47:59.927626 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.927548    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kube-rbac-proxy" containerID="cri-o://0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602" gracePeriod=30
Jul 29 10:47:59.965975 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.965870    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"]
Jul 29 10:47:59.969536 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.969510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:47:59.971446 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.971418    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b12e8-predictor-serving-cert\""
Jul 29 10:47:59.971446 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.971440    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b12e8-kube-rbac-proxy-sar-config\""
Jul 29 10:47:59.978160 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:47:59.978130    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"]
Jul 29 10:48:00.015587 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.015549    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx68f\" (UniqueName: \"kubernetes.io/projected/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-api-access-vx68f\") pod \"success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") " pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.015587 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.015588    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-proxy-tls\") pod \"success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") " pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.015848 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.015618    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") " pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.116970 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.116914    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx68f\" (UniqueName: \"kubernetes.io/projected/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-api-access-vx68f\") pod \"success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") " pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.117150 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.116980    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-proxy-tls\") pod \"success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") " pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.117150 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.117007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") " pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.117719 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.117696    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") " pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.119619 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.119583    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-proxy-tls\") pod \"success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") " pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.123676 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.123649    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx68f\" (UniqueName: \"kubernetes.io/projected/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-api-access-vx68f\") pod \"success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") " pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.284187 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.284136    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.419958 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.419906    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"]
Jul 29 10:48:00.423531 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:48:00.423499    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cca808_2600_4e20_a498_fd3ea5b3c3ce.slice/crio-afce0e0bebf0e70e4c0d690dc0cd57bf3e8db1621d94cdeb4e8d99a015ccd52e WatchSource:0}: Error finding container afce0e0bebf0e70e4c0d690dc0cd57bf3e8db1621d94cdeb4e8d99a015ccd52e: Status 404 returned error can't find the container with id afce0e0bebf0e70e4c0d690dc0cd57bf3e8db1621d94cdeb4e8d99a015ccd52e
Jul 29 10:48:00.723328 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.723292    2572 generic.go:356] "Generic (PLEG): container finished" podID="58ccf183-bc4e-47ee-a353-a4696239b953" containerID="0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602" exitCode=2
Jul 29 10:48:00.723510 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.723370    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" event={"ID":"58ccf183-bc4e-47ee-a353-a4696239b953","Type":"ContainerDied","Data":"0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602"}
Jul 29 10:48:00.725159 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.725125    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" event={"ID":"b6cca808-2600-4e20-a498-fd3ea5b3c3ce","Type":"ContainerStarted","Data":"34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5"}
Jul 29 10:48:00.725303 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.725180    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" event={"ID":"b6cca808-2600-4e20-a498-fd3ea5b3c3ce","Type":"ContainerStarted","Data":"10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df"}
Jul 29 10:48:00.725303 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.725201    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" event={"ID":"b6cca808-2600-4e20-a498-fd3ea5b3c3ce","Type":"ContainerStarted","Data":"afce0e0bebf0e70e4c0d690dc0cd57bf3e8db1621d94cdeb4e8d99a015ccd52e"}
Jul 29 10:48:00.725303 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.725252    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.725303 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.725302    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:00.726672 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.726647    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 29 10:48:00.742105 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:00.742053    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podStartSLOduration=1.7420371129999999 podStartE2EDuration="1.742037113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:48:00.740415871 +0000 UTC m=+1162.968662433" watchObservedRunningTime="2026-07-29 10:48:00.742037113 +0000 UTC m=+1162.970283721"
Jul 29 10:48:01.729003 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:01.728957    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 29 10:48:02.659652 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:02.659601    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 29 10:48:02.666644 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:02.666608    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 29 10:48:02.898439 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:02.898397    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:48:03.283938 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.283902    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:48:03.341173 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.341136    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrs8v\" (UniqueName: \"kubernetes.io/projected/58ccf183-bc4e-47ee-a353-a4696239b953-kube-api-access-vrs8v\") pod \"58ccf183-bc4e-47ee-a353-a4696239b953\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") "
Jul 29 10:48:03.341348 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.341196    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ccf183-bc4e-47ee-a353-a4696239b953-proxy-tls\") pod \"58ccf183-bc4e-47ee-a353-a4696239b953\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") "
Jul 29 10:48:03.341348 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.341226    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ccf183-bc4e-47ee-a353-a4696239b953-kube-rbac-proxy-sar-config\") pod \"58ccf183-bc4e-47ee-a353-a4696239b953\" (UID: \"58ccf183-bc4e-47ee-a353-a4696239b953\") "
Jul 29 10:48:03.341697 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.341652    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ccf183-bc4e-47ee-a353-a4696239b953-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58ccf183-bc4e-47ee-a353-a4696239b953" (UID: "58ccf183-bc4e-47ee-a353-a4696239b953"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:48:03.343312 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.343286    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ccf183-bc4e-47ee-a353-a4696239b953-kube-api-access-vrs8v" (OuterVolumeSpecName: "kube-api-access-vrs8v") pod "58ccf183-bc4e-47ee-a353-a4696239b953" (UID: "58ccf183-bc4e-47ee-a353-a4696239b953"). InnerVolumeSpecName "kube-api-access-vrs8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:48:03.343407 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.343328    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ccf183-bc4e-47ee-a353-a4696239b953-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58ccf183-bc4e-47ee-a353-a4696239b953" (UID: "58ccf183-bc4e-47ee-a353-a4696239b953"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:48:03.441903 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.441855    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrs8v\" (UniqueName: \"kubernetes.io/projected/58ccf183-bc4e-47ee-a353-a4696239b953-kube-api-access-vrs8v\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:48:03.441903 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.441894    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58ccf183-bc4e-47ee-a353-a4696239b953-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:48:03.441903 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.441904    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58ccf183-bc4e-47ee-a353-a4696239b953-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:48:03.738777 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.738682    2572 generic.go:356] "Generic (PLEG): container finished" podID="58ccf183-bc4e-47ee-a353-a4696239b953" containerID="518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7" exitCode=0
Jul 29 10:48:03.738777 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.738740    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"
Jul 29 10:48:03.738777 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.738767    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" event={"ID":"58ccf183-bc4e-47ee-a353-a4696239b953","Type":"ContainerDied","Data":"518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7"}
Jul 29 10:48:03.739034 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.738830    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86" event={"ID":"58ccf183-bc4e-47ee-a353-a4696239b953","Type":"ContainerDied","Data":"1db0a831776fb8cca798b6a7f934e10b1bbeee2d33b7e9de7433ca6775d0fdf1"}
Jul 29 10:48:03.739034 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.738858    2572 scope.go:117] "RemoveContainer" containerID="0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602"
Jul 29 10:48:03.748189 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.748167    2572 scope.go:117] "RemoveContainer" containerID="518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7"
Jul 29 10:48:03.756007 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.755986    2572 scope.go:117] "RemoveContainer" containerID="0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602"
Jul 29 10:48:03.756299 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:48:03.756281    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602\": container with ID starting with 0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602 not found: ID does not exist" containerID="0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602"
Jul 29 10:48:03.756359 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.756308    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602"} err="failed to get container status \"0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602\": rpc error: code = NotFound desc = could not find container \"0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602\": container with ID starting with 0fe45e2ec36ce7b6cfe215d842878d2285a7b282d9f4e54029c64e2542c93602 not found: ID does not exist"
Jul 29 10:48:03.756359 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.756326    2572 scope.go:117] "RemoveContainer" containerID="518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7"
Jul 29 10:48:03.756573 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:48:03.756554    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7\": container with ID starting with 518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7 not found: ID does not exist" containerID="518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7"
Jul 29 10:48:03.756619 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.756580    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7"} err="failed to get container status \"518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7\": rpc error: code = NotFound desc = could not find container \"518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7\": container with ID starting with 518a33e673f5693553b74df62b21854b8984552a919406a52b03c70f1c3716c7 not found: ID does not exist"
Jul 29 10:48:03.759426 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.759395    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"]
Jul 29 10:48:03.763369 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:03.763346    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0a743-predictor-98f985b96-6lt86"]
Jul 29 10:48:04.309193 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:04.309158    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" path="/var/lib/kubelet/pods/58ccf183-bc4e-47ee-a353-a4696239b953/volumes"
Jul 29 10:48:06.735026 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:06.734995    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:06.735512 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:06.735485    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 29 10:48:07.898819 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:07.898771    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:48:12.898329 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:12.898286    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:48:12.898822 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:12.898408    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:48:16.735819 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:16.735777    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 29 10:48:17.898008 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:17.897969    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:48:22.898237 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:22.898198    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:48:26.735473 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:26.735428    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 29 10:48:27.897985 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:27.897945    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:48:29.838519 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:29.838482    2572 generic.go:356] "Generic (PLEG): container finished" podID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerID="383a7c3fda1b23311b81859262d55f529f090ed1a46229a3e43acbd0c0b1303d" exitCode=0
Jul 29 10:48:29.838519 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:29.838524    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" event={"ID":"4d991797-5a7c-439d-af61-5d7f08d8b15a","Type":"ContainerDied","Data":"383a7c3fda1b23311b81859262d55f529f090ed1a46229a3e43acbd0c0b1303d"}
Jul 29 10:48:29.926358 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:29.926330    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:48:30.051069 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.050970    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d991797-5a7c-439d-af61-5d7f08d8b15a-proxy-tls\") pod \"4d991797-5a7c-439d-af61-5d7f08d8b15a\" (UID: \"4d991797-5a7c-439d-af61-5d7f08d8b15a\") "
Jul 29 10:48:30.051069 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.051044    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d991797-5a7c-439d-af61-5d7f08d8b15a-openshift-service-ca-bundle\") pod \"4d991797-5a7c-439d-af61-5d7f08d8b15a\" (UID: \"4d991797-5a7c-439d-af61-5d7f08d8b15a\") "
Jul 29 10:48:30.051409 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.051381    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d991797-5a7c-439d-af61-5d7f08d8b15a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4d991797-5a7c-439d-af61-5d7f08d8b15a" (UID: "4d991797-5a7c-439d-af61-5d7f08d8b15a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:48:30.053154 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.053126    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d991797-5a7c-439d-af61-5d7f08d8b15a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d991797-5a7c-439d-af61-5d7f08d8b15a" (UID: "4d991797-5a7c-439d-af61-5d7f08d8b15a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:48:30.152264 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.152221    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d991797-5a7c-439d-af61-5d7f08d8b15a-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:48:30.152264 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.152260    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d991797-5a7c-439d-af61-5d7f08d8b15a-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:48:30.844115 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.844075    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx" event={"ID":"4d991797-5a7c-439d-af61-5d7f08d8b15a","Type":"ContainerDied","Data":"a40e2c160edf2b54821c00436ac46576e62df638b8b24d7c254e64f97eddc5a6"}
Jul 29 10:48:30.844115 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.844113    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"
Jul 29 10:48:30.844622 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.844148    2572 scope.go:117] "RemoveContainer" containerID="383a7c3fda1b23311b81859262d55f529f090ed1a46229a3e43acbd0c0b1303d"
Jul 29 10:48:30.860754 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.860718    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"]
Jul 29 10:48:30.865010 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:30.864983    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-0a743-9d56587d4-sqlmx"]
Jul 29 10:48:32.309908 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:32.309877    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" path="/var/lib/kubelet/pods/4d991797-5a7c-439d-af61-5d7f08d8b15a/volumes"
Jul 29 10:48:36.735505 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:36.735465    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 29 10:48:46.736156 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:46.736115    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:48:59.585019 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.584972    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"]
Jul 29 10:48:59.585406 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.585192    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40" containerID="cri-o://fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247" gracePeriod=30
Jul 29 10:48:59.733392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.733351    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"]
Jul 29 10:48:59.733682 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.733642    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kube-rbac-proxy" containerID="cri-o://b4b964c0a9b09558ca682ef10e27b3d31267e3e22d10123d551e5cfe576581c0" gracePeriod=30
Jul 29 10:48:59.733914 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.733641    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container" containerID="cri-o://6f8d32337ad6507135892570ef3608a5fc97f5cc20be1b97f4c293af473bf2f9" gracePeriod=30
Jul 29 10:48:59.777569 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777533    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"]
Jul 29 10:48:59.777836 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777824    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743"
Jul 29 10:48:59.777879 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777838    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743"
Jul 29 10:48:59.777879 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777848    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container"
Jul 29 10:48:59.777879 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777854    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container"
Jul 29 10:48:59.777879 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777865    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kube-rbac-proxy"
Jul 29 10:48:59.777879 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777873    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kube-rbac-proxy"
Jul 29 10:48:59.778055 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777952    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d991797-5a7c-439d-af61-5d7f08d8b15a" containerName="switch-graph-0a743"
Jul 29 10:48:59.778055 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777962    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kserve-container"
Jul 29 10:48:59.778055 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.777972    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ccf183-bc4e-47ee-a353-a4696239b953" containerName="kube-rbac-proxy"
Jul 29 10:48:59.782459 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.782437    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:48:59.784544 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.784519    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6242a-predictor-serving-cert\""
Jul 29 10:48:59.784696 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.784524    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6242a-kube-rbac-proxy-sar-config\""
Jul 29 10:48:59.791589 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.791563    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"]
Jul 29 10:48:59.793989 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.793962    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xql2x\" (UniqueName: \"kubernetes.io/projected/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-api-access-xql2x\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:48:59.794133 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.794058    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0afd87a2-5d84-4854-bc44-6835c5be75cb-proxy-tls\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:48:59.794133 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.794088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:48:59.900342 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.894999    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0afd87a2-5d84-4854-bc44-6835c5be75cb-proxy-tls\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:48:59.900342 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.895059    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:48:59.900342 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.895123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xql2x\" (UniqueName: \"kubernetes.io/projected/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-api-access-xql2x\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:48:59.900342 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:48:59.895687    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6242a-predictor-serving-cert: secret "success-200-isvc-6242a-predictor-serving-cert" not found
Jul 29 10:48:59.900342 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:48:59.895791    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0afd87a2-5d84-4854-bc44-6835c5be75cb-proxy-tls podName:0afd87a2-5d84-4854-bc44-6835c5be75cb nodeName:}" failed. No retries permitted until 2026-07-29 10:49:00.395768001 +0000 UTC m=+1222.624014530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0afd87a2-5d84-4854-bc44-6835c5be75cb-proxy-tls") pod "success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" (UID: "0afd87a2-5d84-4854-bc44-6835c5be75cb") : secret "success-200-isvc-6242a-predictor-serving-cert" not found
Jul 29 10:48:59.901258 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.901229    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:48:59.907086 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.907061    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xql2x\" (UniqueName: \"kubernetes.io/projected/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-api-access-xql2x\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:48:59.959408 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.959372    2572 generic.go:356] "Generic (PLEG): container finished" podID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerID="b4b964c0a9b09558ca682ef10e27b3d31267e3e22d10123d551e5cfe576581c0" exitCode=2
Jul 29 10:48:59.959585 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:48:59.959443    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" event={"ID":"d7ca144f-8def-4bac-a45f-89d3695e9928","Type":"ContainerDied","Data":"b4b964c0a9b09558ca682ef10e27b3d31267e3e22d10123d551e5cfe576581c0"}
Jul 29 10:49:00.030035 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.029993    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"]
Jul 29 10:49:00.033383 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.033363    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:00.035577 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.035554    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b12e8-serving-cert\""
Jul 29 10:49:00.035711 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.035593    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b12e8-kube-rbac-proxy-sar-config\""
Jul 29 10:49:00.043668 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.043639    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"]
Jul 29 10:49:00.097241 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.097201    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-openshift-service-ca-bundle\") pod \"ensemble-graph-b12e8-db4cf88d6-fzbjk\" (UID: \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\") " pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:00.097413 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.097259    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-proxy-tls\") pod \"ensemble-graph-b12e8-db4cf88d6-fzbjk\" (UID: \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\") " pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:00.198236 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.198197    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-openshift-service-ca-bundle\") pod \"ensemble-graph-b12e8-db4cf88d6-fzbjk\" (UID: \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\") " pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:00.198236 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.198242    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-proxy-tls\") pod \"ensemble-graph-b12e8-db4cf88d6-fzbjk\" (UID: \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\") " pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:00.198478 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:49:00.198346    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-b12e8-serving-cert: secret "ensemble-graph-b12e8-serving-cert" not found
Jul 29 10:49:00.198478 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:49:00.198400    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-proxy-tls podName:8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3 nodeName:}" failed. No retries permitted until 2026-07-29 10:49:00.69838524 +0000 UTC m=+1222.926631769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-proxy-tls") pod "ensemble-graph-b12e8-db4cf88d6-fzbjk" (UID: "8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3") : secret "ensemble-graph-b12e8-serving-cert" not found
Jul 29 10:49:00.199032 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.199007    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-openshift-service-ca-bundle\") pod \"ensemble-graph-b12e8-db4cf88d6-fzbjk\" (UID: \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\") " pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:00.400227 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.400181    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0afd87a2-5d84-4854-bc44-6835c5be75cb-proxy-tls\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:49:00.402844 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.402803    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0afd87a2-5d84-4854-bc44-6835c5be75cb-proxy-tls\") pod \"success-200-isvc-6242a-predictor-cbf7db67b-9cr5q\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") " pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:49:00.694020 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.693976    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:49:00.702294 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.702247    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-proxy-tls\") pod \"ensemble-graph-b12e8-db4cf88d6-fzbjk\" (UID: \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\") " pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:00.704652 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.704622    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-proxy-tls\") pod \"ensemble-graph-b12e8-db4cf88d6-fzbjk\" (UID: \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\") " pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:00.832987 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.832947    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"]
Jul 29 10:49:00.835908 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:49:00.835879    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afd87a2_5d84_4854_bc44_6835c5be75cb.slice/crio-986e82afd2365d0a842258ac058726736fe84a3308616b73aff8097e1afb48c9 WatchSource:0}: Error finding container 986e82afd2365d0a842258ac058726736fe84a3308616b73aff8097e1afb48c9: Status 404 returned error can't find the container with id 986e82afd2365d0a842258ac058726736fe84a3308616b73aff8097e1afb48c9
Jul 29 10:49:00.948031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.948003    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:00.965904 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.965869    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" event={"ID":"0afd87a2-5d84-4854-bc44-6835c5be75cb","Type":"ContainerStarted","Data":"951dea11166ca2f30d26454b1066370a4d1d3fde62bcdd9de49e2d4f66bb9fc5"}
Jul 29 10:49:00.966081 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.965968    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" event={"ID":"0afd87a2-5d84-4854-bc44-6835c5be75cb","Type":"ContainerStarted","Data":"decdbd4a0c543a3e716a5ccdc2cc09e41d0118304551c43b19323ebad7024fee"}
Jul 29 10:49:00.966081 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.966003    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" event={"ID":"0afd87a2-5d84-4854-bc44-6835c5be75cb","Type":"ContainerStarted","Data":"986e82afd2365d0a842258ac058726736fe84a3308616b73aff8097e1afb48c9"}
Jul 29 10:49:00.966081 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.966044    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:49:00.986250 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:00.986185    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podStartSLOduration=1.986158246 podStartE2EDuration="1.986158246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:49:00.984636661 +0000 UTC m=+1223.212883211" watchObservedRunningTime="2026-07-29 10:49:00.986158246 +0000 UTC m=+1223.214404797"
Jul 29 10:49:01.091516 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:01.091478    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"]
Jul 29 10:49:01.094828 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:49:01.094800    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8416c8b6_3d27_4e6b_8a17_e106ec8aa4c3.slice/crio-6891aef8b37f84f312c1889b8b9d17feaf17d04d3a2e9e84f2686fd0fb5dada2 WatchSource:0}: Error finding container 6891aef8b37f84f312c1889b8b9d17feaf17d04d3a2e9e84f2686fd0fb5dada2: Status 404 returned error can't find the container with id 6891aef8b37f84f312c1889b8b9d17feaf17d04d3a2e9e84f2686fd0fb5dada2
Jul 29 10:49:01.972239 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:01.972196    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" event={"ID":"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3","Type":"ContainerStarted","Data":"e1b11acfc9f3473b95c83fde57cdaa23c1b42917d7ddb9c868b178b42c269aca"}
Jul 29 10:49:01.972659 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:01.972260    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:01.972659 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:01.972276    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" event={"ID":"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3","Type":"ContainerStarted","Data":"6891aef8b37f84f312c1889b8b9d17feaf17d04d3a2e9e84f2686fd0fb5dada2"}
Jul 29 10:49:01.972659 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:01.972544    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:49:01.973899 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:01.973870    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 29 10:49:01.988585 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:01.988534    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" podStartSLOduration=1.988515005 podStartE2EDuration="1.988515005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:49:01.987388606 +0000 UTC m=+1224.215635156" watchObservedRunningTime="2026-07-29 10:49:01.988515005 +0000 UTC m=+1224.216761555"
Jul 29 10:49:02.121872 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:02.121823    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:02.978283 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:02.978246    2572 generic.go:356] "Generic (PLEG): container finished" podID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerID="6f8d32337ad6507135892570ef3608a5fc97f5cc20be1b97f4c293af473bf2f9" exitCode=0
Jul 29 10:49:02.978744 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:02.978311    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" event={"ID":"d7ca144f-8def-4bac-a45f-89d3695e9928","Type":"ContainerDied","Data":"6f8d32337ad6507135892570ef3608a5fc97f5cc20be1b97f4c293af473bf2f9"}
Jul 29 10:49:02.978744 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:02.978715    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 29 10:49:02.996390 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:02.996363    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:49:03.020147 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.020114    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls\") pod \"d7ca144f-8def-4bac-a45f-89d3695e9928\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") "
Jul 29 10:49:03.020330 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.020179    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-897sq\" (UniqueName: \"kubernetes.io/projected/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-api-access-897sq\") pod \"d7ca144f-8def-4bac-a45f-89d3695e9928\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") "
Jul 29 10:49:03.020330 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.020227    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-rbac-proxy-sar-config\") pod \"d7ca144f-8def-4bac-a45f-89d3695e9928\" (UID: \"d7ca144f-8def-4bac-a45f-89d3695e9928\") "
Jul 29 10:49:03.020847 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.020811    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7ca144f-8def-4bac-a45f-89d3695e9928" (UID: "d7ca144f-8def-4bac-a45f-89d3695e9928"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:49:03.022391 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.022358    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7ca144f-8def-4bac-a45f-89d3695e9928" (UID: "d7ca144f-8def-4bac-a45f-89d3695e9928"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:49:03.022750 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.022728    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-api-access-897sq" (OuterVolumeSpecName: "kube-api-access-897sq") pod "d7ca144f-8def-4bac-a45f-89d3695e9928" (UID: "d7ca144f-8def-4bac-a45f-89d3695e9928"). InnerVolumeSpecName "kube-api-access-897sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:49:03.121830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.121798    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7ca144f-8def-4bac-a45f-89d3695e9928-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:03.121830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.121831    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-897sq\" (UniqueName: \"kubernetes.io/projected/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-api-access-897sq\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:03.122066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.121843    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7ca144f-8def-4bac-a45f-89d3695e9928-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:03.983468 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.983431    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs" event={"ID":"d7ca144f-8def-4bac-a45f-89d3695e9928","Type":"ContainerDied","Data":"26b8013f2b7aac73b06b9b2682fcffa793bc1c3b2c86be05735dc9b11ca7318f"}
Jul 29 10:49:03.983943 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.983485    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"
Jul 29 10:49:03.983943 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.983511    2572 scope.go:117] "RemoveContainer" containerID="b4b964c0a9b09558ca682ef10e27b3d31267e3e22d10123d551e5cfe576581c0"
Jul 29 10:49:03.992207 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:03.992188    2572 scope.go:117] "RemoveContainer" containerID="6f8d32337ad6507135892570ef3608a5fc97f5cc20be1b97f4c293af473bf2f9"
Jul 29 10:49:04.004384 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:04.004353    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"]
Jul 29 10:49:04.006099 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:04.006078    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f1d40-predictor-58f666b764-89zbs"]
Jul 29 10:49:04.309764 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:04.309684    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" path="/var/lib/kubelet/pods/d7ca144f-8def-4bac-a45f-89d3695e9928/volumes"
Jul 29 10:49:07.122639 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:07.122594    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:07.983185 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:07.983149    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:49:07.983717 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:07.983687    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 29 10:49:07.984745 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:07.984721    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:10.101991 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.101876    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"]
Jul 29 10:49:10.102441 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.102120    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8" containerID="cri-o://e1b11acfc9f3473b95c83fde57cdaa23c1b42917d7ddb9c868b178b42c269aca" gracePeriod=30
Jul 29 10:49:10.240092 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.240056    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"]
Jul 29 10:49:10.240351 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.240305    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container" containerID="cri-o://10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df" gracePeriod=30
Jul 29 10:49:10.240351 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.240334    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kube-rbac-proxy" containerID="cri-o://34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5" gracePeriod=30
Jul 29 10:49:10.260965 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.260899    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"]
Jul 29 10:49:10.261230 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.261217    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container"
Jul 29 10:49:10.261272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.261233    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container"
Jul 29 10:49:10.261272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.261245    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kube-rbac-proxy"
Jul 29 10:49:10.261272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.261252    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kube-rbac-proxy"
Jul 29 10:49:10.261364 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.261306    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kserve-container"
Jul 29 10:49:10.261364 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.261315    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ca144f-8def-4bac-a45f-89d3695e9928" containerName="kube-rbac-proxy"
Jul 29 10:49:10.264140 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.264121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.266720 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.266691    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d2d23-predictor-serving-cert\""
Jul 29 10:49:10.266884 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.266792    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d2d23-kube-rbac-proxy-sar-config\""
Jul 29 10:49:10.275246 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.275208    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"]
Jul 29 10:49:10.276261 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.276228    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d53f76-1981-4b72-b824-2c949abdf491-proxy-tls\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.276389 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.276273    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d53f76-1981-4b72-b824-2c949abdf491-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.276389 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.276328    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgrs\" (UniqueName: \"kubernetes.io/projected/44d53f76-1981-4b72-b824-2c949abdf491-kube-api-access-nmgrs\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.377513 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.377470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgrs\" (UniqueName: \"kubernetes.io/projected/44d53f76-1981-4b72-b824-2c949abdf491-kube-api-access-nmgrs\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.377741 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.377587    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d53f76-1981-4b72-b824-2c949abdf491-proxy-tls\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.377741 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.377631    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d53f76-1981-4b72-b824-2c949abdf491-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.377872 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:49:10.377740    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-serving-cert: secret "success-200-isvc-d2d23-predictor-serving-cert" not found
Jul 29 10:49:10.377872 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:49:10.377818    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44d53f76-1981-4b72-b824-2c949abdf491-proxy-tls podName:44d53f76-1981-4b72-b824-2c949abdf491 nodeName:}" failed. No retries permitted until 2026-07-29 10:49:10.877793371 +0000 UTC m=+1233.106039919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/44d53f76-1981-4b72-b824-2c949abdf491-proxy-tls") pod "success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" (UID: "44d53f76-1981-4b72-b824-2c949abdf491") : secret "success-200-isvc-d2d23-predictor-serving-cert" not found
Jul 29 10:49:10.378382 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.378361    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d53f76-1981-4b72-b824-2c949abdf491-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.389376 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.389341    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgrs\" (UniqueName: \"kubernetes.io/projected/44d53f76-1981-4b72-b824-2c949abdf491-kube-api-access-nmgrs\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.880533 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.880488    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d53f76-1981-4b72-b824-2c949abdf491-proxy-tls\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:10.883009 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:10.882973    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d53f76-1981-4b72-b824-2c949abdf491-proxy-tls\") pod \"success-200-isvc-d2d23-predictor-77b7bd766d-vx77s\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") " pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:11.012025 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:11.011992    2572 generic.go:356] "Generic (PLEG): container finished" podID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerID="34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5" exitCode=2
Jul 29 10:49:11.012218 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:11.012068    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" event={"ID":"b6cca808-2600-4e20-a498-fd3ea5b3c3ce","Type":"ContainerDied","Data":"34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5"}
Jul 29 10:49:11.174275 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:11.174192    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:11.317603 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:11.317567    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"]
Jul 29 10:49:11.320813 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:49:11.320782    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d53f76_1981_4b72_b824_2c949abdf491.slice/crio-aed62492e1b313a189ce30742b39ccb839623b383fd43976ff5e24a803965e78 WatchSource:0}: Error finding container aed62492e1b313a189ce30742b39ccb839623b383fd43976ff5e24a803965e78: Status 404 returned error can't find the container with id aed62492e1b313a189ce30742b39ccb839623b383fd43976ff5e24a803965e78
Jul 29 10:49:11.729461 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:11.729341    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 29 10:49:12.018581 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.018491    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" event={"ID":"44d53f76-1981-4b72-b824-2c949abdf491","Type":"ContainerStarted","Data":"8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148"}
Jul 29 10:49:12.018581 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.018552    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" event={"ID":"44d53f76-1981-4b72-b824-2c949abdf491","Type":"ContainerStarted","Data":"768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70"}
Jul 29 10:49:12.018581 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.018576    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" event={"ID":"44d53f76-1981-4b72-b824-2c949abdf491","Type":"ContainerStarted","Data":"aed62492e1b313a189ce30742b39ccb839623b383fd43976ff5e24a803965e78"}
Jul 29 10:49:12.018855 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.018681    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:12.018855 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.018730    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:12.020175 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.020147    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 29 10:49:12.036205 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.036151    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podStartSLOduration=2.036132457 podStartE2EDuration="2.036132457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:49:12.035100863 +0000 UTC m=+1234.263347412" watchObservedRunningTime="2026-07-29 10:49:12.036132457 +0000 UTC m=+1234.264379007"
Jul 29 10:49:12.122118 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.122077    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:12.122290 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.122185    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:49:12.984986 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:12.984949    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:13.028336 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.028293    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 29 10:49:13.597113 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.597083    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:49:13.700817 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.700719    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-rbac-proxy-sar-config\") pod \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") "
Jul 29 10:49:13.700982 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.700819    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-proxy-tls\") pod \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") "
Jul 29 10:49:13.700982 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.700842    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx68f\" (UniqueName: \"kubernetes.io/projected/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-api-access-vx68f\") pod \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\" (UID: \"b6cca808-2600-4e20-a498-fd3ea5b3c3ce\") "
Jul 29 10:49:13.701110 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.701083    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6cca808-2600-4e20-a498-fd3ea5b3c3ce" (UID: "b6cca808-2600-4e20-a498-fd3ea5b3c3ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:49:13.703059 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.703026    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-api-access-vx68f" (OuterVolumeSpecName: "kube-api-access-vx68f") pod "b6cca808-2600-4e20-a498-fd3ea5b3c3ce" (UID: "b6cca808-2600-4e20-a498-fd3ea5b3c3ce"). InnerVolumeSpecName "kube-api-access-vx68f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:49:13.703186 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.703097    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6cca808-2600-4e20-a498-fd3ea5b3c3ce" (UID: "b6cca808-2600-4e20-a498-fd3ea5b3c3ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:49:13.802076 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.802039    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:13.802076 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.802068    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx68f\" (UniqueName: \"kubernetes.io/projected/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-api-access-vx68f\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:13.802076 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:13.802078    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6cca808-2600-4e20-a498-fd3ea5b3c3ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:14.033878 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.033783    2572 generic.go:356] "Generic (PLEG): container finished" podID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerID="10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df" exitCode=0
Jul 29 10:49:14.033878 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.033836    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"
Jul 29 10:49:14.034457 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.033873    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" event={"ID":"b6cca808-2600-4e20-a498-fd3ea5b3c3ce","Type":"ContainerDied","Data":"10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df"}
Jul 29 10:49:14.034457 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.033971    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw" event={"ID":"b6cca808-2600-4e20-a498-fd3ea5b3c3ce","Type":"ContainerDied","Data":"afce0e0bebf0e70e4c0d690dc0cd57bf3e8db1621d94cdeb4e8d99a015ccd52e"}
Jul 29 10:49:14.034457 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.034013    2572 scope.go:117] "RemoveContainer" containerID="34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5"
Jul 29 10:49:14.042318 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.042298    2572 scope.go:117] "RemoveContainer" containerID="10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df"
Jul 29 10:49:14.049639 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.049618    2572 scope.go:117] "RemoveContainer" containerID="34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5"
Jul 29 10:49:14.049931 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:49:14.049893    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5\": container with ID starting with 34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5 not found: ID does not exist" containerID="34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5"
Jul 29 10:49:14.050026 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.049944    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5"} err="failed to get container status \"34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5\": rpc error: code = NotFound desc = could not find container \"34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5\": container with ID starting with 34134a433ef2d1a0d0888788edfb166c3cbd421c1b1f2a6475cfd2e8421ed2f5 not found: ID does not exist"
Jul 29 10:49:14.050026 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.049970    2572 scope.go:117] "RemoveContainer" containerID="10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df"
Jul 29 10:49:14.050556 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:49:14.050538    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df\": container with ID starting with 10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df not found: ID does not exist" containerID="10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df"
Jul 29 10:49:14.050595 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.050561    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df"} err="failed to get container status \"10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df\": rpc error: code = NotFound desc = could not find container \"10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df\": container with ID starting with 10e69acf24d0dfcbe873214442d9dde0424ca5dbcf02192670821d52e90475df not found: ID does not exist"
Jul 29 10:49:14.053914 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.053891    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"]
Jul 29 10:49:14.056058 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.056034    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b12e8-predictor-57c478d4fd-lwqkw"]
Jul 29 10:49:14.309765 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:14.309675    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" path="/var/lib/kubelet/pods/b6cca808-2600-4e20-a498-fd3ea5b3c3ce/volumes"
Jul 29 10:49:17.121860 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:17.121821    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:17.983455 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:17.983410    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:17.983650 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:17.983628    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 29 10:49:18.032786 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:18.032755    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:18.033376 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:18.033347    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 29 10:49:22.121600 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:22.121561    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:22.982900 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:22.982854    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:22.983115 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:22.982989    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:27.121887 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:27.121845    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:27.982888 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:27.982840    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:27.984446 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:27.984417    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 29 10:49:28.033844 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:28.033805    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 29 10:49:29.736879 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:29.736850    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:49:29.825343 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:29.825293    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/772851ff-213c-4341-9eeb-85bcbe08cf6c-openshift-service-ca-bundle\") pod \"772851ff-213c-4341-9eeb-85bcbe08cf6c\" (UID: \"772851ff-213c-4341-9eeb-85bcbe08cf6c\") "
Jul 29 10:49:29.825561 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:29.825403    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772851ff-213c-4341-9eeb-85bcbe08cf6c-proxy-tls\") pod \"772851ff-213c-4341-9eeb-85bcbe08cf6c\" (UID: \"772851ff-213c-4341-9eeb-85bcbe08cf6c\") "
Jul 29 10:49:29.825666 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:29.825645    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772851ff-213c-4341-9eeb-85bcbe08cf6c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "772851ff-213c-4341-9eeb-85bcbe08cf6c" (UID: "772851ff-213c-4341-9eeb-85bcbe08cf6c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:49:29.827542 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:29.827512    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772851ff-213c-4341-9eeb-85bcbe08cf6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "772851ff-213c-4341-9eeb-85bcbe08cf6c" (UID: "772851ff-213c-4341-9eeb-85bcbe08cf6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:49:29.926266 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:29.926225    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772851ff-213c-4341-9eeb-85bcbe08cf6c-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:29.926266 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:29.926256    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/772851ff-213c-4341-9eeb-85bcbe08cf6c-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:30.096148 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.096065    2572 generic.go:356] "Generic (PLEG): container finished" podID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerID="fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247" exitCode=0
Jul 29 10:49:30.096148 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.096112    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"
Jul 29 10:49:30.096148 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.096121    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" event={"ID":"772851ff-213c-4341-9eeb-85bcbe08cf6c","Type":"ContainerDied","Data":"fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247"}
Jul 29 10:49:30.096392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.096169    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw" event={"ID":"772851ff-213c-4341-9eeb-85bcbe08cf6c","Type":"ContainerDied","Data":"f948df69ce71f7951f985fa97d2f0eb1ccdb5baf1bffa1025d872077ddf66d2a"}
Jul 29 10:49:30.096392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.096202    2572 scope.go:117] "RemoveContainer" containerID="fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247"
Jul 29 10:49:30.104407 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.104385    2572 scope.go:117] "RemoveContainer" containerID="fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247"
Jul 29 10:49:30.104744 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:49:30.104722    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247\": container with ID starting with fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247 not found: ID does not exist" containerID="fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247"
Jul 29 10:49:30.104780 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.104754    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247"} err="failed to get container status \"fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247\": rpc error: code = NotFound desc = could not find container \"fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247\": container with ID starting with fb4467abcc0dac9d0e006f0c2257bbf75a1a55d2812ba131a08206dcc803d247 not found: ID does not exist"
Jul 29 10:49:30.115851 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.115824    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"]
Jul 29 10:49:30.118261 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.118236    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f1d40-78bdff7fd9-5sdnw"]
Jul 29 10:49:30.309431 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:30.309339    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" path="/var/lib/kubelet/pods/772851ff-213c-4341-9eeb-85bcbe08cf6c/volumes"
Jul 29 10:49:32.982673 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:32.982627    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:37.982658 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:37.982617    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:49:37.983867 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:37.983840    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 29 10:49:38.034130 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:38.034085    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 29 10:49:40.137664 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:40.137623    2572 generic.go:356] "Generic (PLEG): container finished" podID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerID="e1b11acfc9f3473b95c83fde57cdaa23c1b42917d7ddb9c868b178b42c269aca" exitCode=0
Jul 29 10:49:40.138058 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:40.137692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" event={"ID":"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3","Type":"ContainerDied","Data":"e1b11acfc9f3473b95c83fde57cdaa23c1b42917d7ddb9c868b178b42c269aca"}
Jul 29 10:49:40.258237 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:40.258210    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:40.308488 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:40.308452    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-proxy-tls\") pod \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\" (UID: \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\") "
Jul 29 10:49:40.308697 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:40.308509    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-openshift-service-ca-bundle\") pod \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\" (UID: \"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3\") "
Jul 29 10:49:40.308914 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:40.308892    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" (UID: "8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:49:40.310643 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:40.310605    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" (UID: "8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:49:40.410079 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:40.410034    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:40.410079 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:40.410074    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:49:41.143072 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:41.143031    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk" event={"ID":"8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3","Type":"ContainerDied","Data":"6891aef8b37f84f312c1889b8b9d17feaf17d04d3a2e9e84f2686fd0fb5dada2"}
Jul 29 10:49:41.143072 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:41.143073    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"
Jul 29 10:49:41.143626 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:41.143131    2572 scope.go:117] "RemoveContainer" containerID="e1b11acfc9f3473b95c83fde57cdaa23c1b42917d7ddb9c868b178b42c269aca"
Jul 29 10:49:41.164901 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:41.164869    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"]
Jul 29 10:49:41.168697 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:41.168670    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b12e8-db4cf88d6-fzbjk"]
Jul 29 10:49:42.310288 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:42.310247    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" path="/var/lib/kubelet/pods/8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3/volumes"
Jul 29 10:49:47.985139 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:47.985109    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:49:48.033885 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:48.033828    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 29 10:49:58.034626 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:58.034596    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:49:59.813680 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.813646    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"]
Jul 29 10:49:59.814066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.813943    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kube-rbac-proxy"
Jul 29 10:49:59.814066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.813958    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kube-rbac-proxy"
Jul 29 10:49:59.814066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.813977    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40"
Jul 29 10:49:59.814066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.813983    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40"
Jul 29 10:49:59.814066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.813994    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8"
Jul 29 10:49:59.814066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.814025    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8"
Jul 29 10:49:59.814066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.814037    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container"
Jul 29 10:49:59.814066 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.814041    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container"
Jul 29 10:49:59.814311 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.814095    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="772851ff-213c-4341-9eeb-85bcbe08cf6c" containerName="sequence-graph-f1d40"
Jul 29 10:49:59.814311 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.814106    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kserve-container"
Jul 29 10:49:59.814311 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.814113    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8416c8b6-3d27-4e6b-8a17-e106ec8aa4c3" containerName="ensemble-graph-b12e8"
Jul 29 10:49:59.814311 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.814120    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cca808-2600-4e20-a498-fd3ea5b3c3ce" containerName="kube-rbac-proxy"
Jul 29 10:49:59.818237 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.818216    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:49:59.820158 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.820135    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-6242a-serving-cert\""
Jul 29 10:49:59.820310 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.820202    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-6242a-kube-rbac-proxy-sar-config\""
Jul 29 10:49:59.828333 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.828304    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"]
Jul 29 10:49:59.966891 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.966839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-proxy-tls\") pod \"sequence-graph-6242a-6df9fc859f-jpk45\" (UID: \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\") " pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:49:59.966891 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:49:59.966896    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-openshift-service-ca-bundle\") pod \"sequence-graph-6242a-6df9fc859f-jpk45\" (UID: \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\") " pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:00.067645 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:00.067531    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-proxy-tls\") pod \"sequence-graph-6242a-6df9fc859f-jpk45\" (UID: \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\") " pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:00.067645 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:00.067587    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-openshift-service-ca-bundle\") pod \"sequence-graph-6242a-6df9fc859f-jpk45\" (UID: \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\") " pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:00.067855 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:50:00.067703    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-6242a-serving-cert: secret "sequence-graph-6242a-serving-cert" not found
Jul 29 10:50:00.067855 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:50:00.067772    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-proxy-tls podName:c10b7ba0-85a3-4dd2-a8b6-3c9809084551 nodeName:}" failed. No retries permitted until 2026-07-29 10:50:00.567754314 +0000 UTC m=+1282.796000843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-proxy-tls") pod "sequence-graph-6242a-6df9fc859f-jpk45" (UID: "c10b7ba0-85a3-4dd2-a8b6-3c9809084551") : secret "sequence-graph-6242a-serving-cert" not found
Jul 29 10:50:00.068217 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:00.068200    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-openshift-service-ca-bundle\") pod \"sequence-graph-6242a-6df9fc859f-jpk45\" (UID: \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\") " pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:00.572038 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:00.571990    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-proxy-tls\") pod \"sequence-graph-6242a-6df9fc859f-jpk45\" (UID: \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\") " pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:00.574551 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:00.574520    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-proxy-tls\") pod \"sequence-graph-6242a-6df9fc859f-jpk45\" (UID: \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\") " pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:00.729272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:00.729231    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:00.865402 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:00.865305    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"]
Jul 29 10:50:00.869999 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:50:00.869964    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10b7ba0_85a3_4dd2_a8b6_3c9809084551.slice/crio-867d5c1705ae7a36afc667176034b67757bfffd5cb197fc896933aa88a7042ae WatchSource:0}: Error finding container 867d5c1705ae7a36afc667176034b67757bfffd5cb197fc896933aa88a7042ae: Status 404 returned error can't find the container with id 867d5c1705ae7a36afc667176034b67757bfffd5cb197fc896933aa88a7042ae
Jul 29 10:50:01.219240 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:01.219197    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" event={"ID":"c10b7ba0-85a3-4dd2-a8b6-3c9809084551","Type":"ContainerStarted","Data":"eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510"}
Jul 29 10:50:01.219429 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:01.219292    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" event={"ID":"c10b7ba0-85a3-4dd2-a8b6-3c9809084551","Type":"ContainerStarted","Data":"867d5c1705ae7a36afc667176034b67757bfffd5cb197fc896933aa88a7042ae"}
Jul 29 10:50:01.219429 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:01.219320    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:01.236687 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:01.236627    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" podStartSLOduration=2.236606158 podStartE2EDuration="2.236606158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:50:01.235553265 +0000 UTC m=+1283.463799828" watchObservedRunningTime="2026-07-29 10:50:01.236606158 +0000 UTC m=+1283.464852710"
Jul 29 10:50:07.229207 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:07.229178    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:09.887255 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:09.887197    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"]
Jul 29 10:50:09.887746 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:09.887478    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a" containerID="cri-o://eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510" gracePeriod=30
Jul 29 10:50:09.996605 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:09.996564    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"]
Jul 29 10:50:09.997087 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:09.996899    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container" containerID="cri-o://decdbd4a0c543a3e716a5ccdc2cc09e41d0118304551c43b19323ebad7024fee" gracePeriod=30
Jul 29 10:50:09.997087 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:09.996910    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kube-rbac-proxy" containerID="cri-o://951dea11166ca2f30d26454b1066370a4d1d3fde62bcdd9de49e2d4f66bb9fc5" gracePeriod=30
Jul 29 10:50:10.050284 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.050247    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"]
Jul 29 10:50:10.053806 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.053780    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.055916 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.055886    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0934b-kube-rbac-proxy-sar-config\""
Jul 29 10:50:10.056091 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.055886    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0934b-predictor-serving-cert\""
Jul 29 10:50:10.063219 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.063189    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"]
Jul 29 10:50:10.150169 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.150113    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckw8m\" (UniqueName: \"kubernetes.io/projected/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-api-access-ckw8m\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.150361 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.150184    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.150361 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.150234    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-proxy-tls\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.250628 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.250590    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-proxy-tls\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.250830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.250688    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckw8m\" (UniqueName: \"kubernetes.io/projected/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-api-access-ckw8m\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.250830 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.250741    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.250830 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:50:10.250763    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-0934b-predictor-serving-cert: secret "success-200-isvc-0934b-predictor-serving-cert" not found
Jul 29 10:50:10.251039 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:50:10.250835    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-proxy-tls podName:cded21d0-08a5-4c0d-916b-b27ccc2d6ccd nodeName:}" failed. No retries permitted until 2026-07-29 10:50:10.750815044 +0000 UTC m=+1292.979061571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-proxy-tls") pod "success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" (UID: "cded21d0-08a5-4c0d-916b-b27ccc2d6ccd") : secret "success-200-isvc-0934b-predictor-serving-cert" not found
Jul 29 10:50:10.251464 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.251444    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.255675 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.255644    2572 generic.go:356] "Generic (PLEG): container finished" podID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerID="951dea11166ca2f30d26454b1066370a4d1d3fde62bcdd9de49e2d4f66bb9fc5" exitCode=2
Jul 29 10:50:10.255822 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.255684    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" event={"ID":"0afd87a2-5d84-4854-bc44-6835c5be75cb","Type":"ContainerDied","Data":"951dea11166ca2f30d26454b1066370a4d1d3fde62bcdd9de49e2d4f66bb9fc5"}
Jul 29 10:50:10.261046 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.260983    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckw8m\" (UniqueName: \"kubernetes.io/projected/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-api-access-ckw8m\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.311129 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.311099    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"]
Jul 29 10:50:10.314342 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.314326    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:10.316449 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.316423    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d2d23-serving-cert\""
Jul 29 10:50:10.316568 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.316422    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d2d23-kube-rbac-proxy-sar-config\""
Jul 29 10:50:10.321158 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.321131    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"]
Jul 29 10:50:10.451898 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.451860    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-openshift-service-ca-bundle\") pod \"ensemble-graph-d2d23-557d84f99-bwkd2\" (UID: \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\") " pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:10.452097 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.451955    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-proxy-tls\") pod \"ensemble-graph-d2d23-557d84f99-bwkd2\" (UID: \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\") " pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:10.553453 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.553348    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-proxy-tls\") pod \"ensemble-graph-d2d23-557d84f99-bwkd2\" (UID: \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\") " pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:10.553596 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.553471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-openshift-service-ca-bundle\") pod \"ensemble-graph-d2d23-557d84f99-bwkd2\" (UID: \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\") " pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:10.553596 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:50:10.553507    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-d2d23-serving-cert: secret "ensemble-graph-d2d23-serving-cert" not found
Jul 29 10:50:10.553596 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:50:10.553590    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-proxy-tls podName:f8bc01a7-7322-4593-92ff-5f0e5e9a4e12 nodeName:}" failed. No retries permitted until 2026-07-29 10:50:11.053569908 +0000 UTC m=+1293.281816444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-proxy-tls") pod "ensemble-graph-d2d23-557d84f99-bwkd2" (UID: "f8bc01a7-7322-4593-92ff-5f0e5e9a4e12") : secret "ensemble-graph-d2d23-serving-cert" not found
Jul 29 10:50:10.554129 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.554112    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-openshift-service-ca-bundle\") pod \"ensemble-graph-d2d23-557d84f99-bwkd2\" (UID: \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\") " pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:10.754914 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.754870    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-proxy-tls\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.757395 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.757367    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-proxy-tls\") pod \"success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") " pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:10.966633 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:10.966589    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:11.057489 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.057456    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-proxy-tls\") pod \"ensemble-graph-d2d23-557d84f99-bwkd2\" (UID: \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\") " pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:11.060366 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.060327    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-proxy-tls\") pod \"ensemble-graph-d2d23-557d84f99-bwkd2\" (UID: \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\") " pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:11.107155 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.106913    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"]
Jul 29 10:50:11.109809 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:50:11.109774    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcded21d0_08a5_4c0d_916b_b27ccc2d6ccd.slice/crio-127fc2a1bd055e2ff5d1eeaa8edc308295bae3261703dd08fac82dcd1d37742f WatchSource:0}: Error finding container 127fc2a1bd055e2ff5d1eeaa8edc308295bae3261703dd08fac82dcd1d37742f: Status 404 returned error can't find the container with id 127fc2a1bd055e2ff5d1eeaa8edc308295bae3261703dd08fac82dcd1d37742f
Jul 29 10:50:11.227361 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.227330    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:11.261547 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.261510    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" event={"ID":"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd","Type":"ContainerStarted","Data":"a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9"}
Jul 29 10:50:11.261754 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.261582    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" event={"ID":"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd","Type":"ContainerStarted","Data":"0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7"}
Jul 29 10:50:11.261754 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.261617    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:11.261754 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.261644    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:11.261754 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.261667    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" event={"ID":"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd","Type":"ContainerStarted","Data":"127fc2a1bd055e2ff5d1eeaa8edc308295bae3261703dd08fac82dcd1d37742f"}
Jul 29 10:50:11.263276 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.263241    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 29 10:50:11.280268 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.280195    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podStartSLOduration=1.280169345 podStartE2EDuration="1.280169345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:50:11.279146985 +0000 UTC m=+1293.507393542" watchObservedRunningTime="2026-07-29 10:50:11.280169345 +0000 UTC m=+1293.508415895"
Jul 29 10:50:11.376248 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:11.376214    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"]
Jul 29 10:50:11.379334 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:50:11.379306    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bc01a7_7322_4593_92ff_5f0e5e9a4e12.slice/crio-ead4b7f2ddbb2e37915c7660ab13e1c9b61e04a107b2b686953382aa0a676db6 WatchSource:0}: Error finding container ead4b7f2ddbb2e37915c7660ab13e1c9b61e04a107b2b686953382aa0a676db6: Status 404 returned error can't find the container with id ead4b7f2ddbb2e37915c7660ab13e1c9b61e04a107b2b686953382aa0a676db6
Jul 29 10:50:12.227502 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:12.227461    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:50:12.266678 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:12.266639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" event={"ID":"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12","Type":"ContainerStarted","Data":"97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea"}
Jul 29 10:50:12.266887 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:12.266703    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" event={"ID":"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12","Type":"ContainerStarted","Data":"ead4b7f2ddbb2e37915c7660ab13e1c9b61e04a107b2b686953382aa0a676db6"}
Jul 29 10:50:12.266887 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:12.266725    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:12.267056 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:12.267032    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 29 10:50:12.285026 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:12.284961    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" podStartSLOduration=2.284942508 podStartE2EDuration="2.284942508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:50:12.283530574 +0000 UTC m=+1294.511777125" watchObservedRunningTime="2026-07-29 10:50:12.284942508 +0000 UTC m=+1294.513189075"
Jul 29 10:50:12.980021 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:12.979972    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 29 10:50:13.273103 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.273066    2572 generic.go:356] "Generic (PLEG): container finished" podID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerID="decdbd4a0c543a3e716a5ccdc2cc09e41d0118304551c43b19323ebad7024fee" exitCode=0
Jul 29 10:50:13.273495 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.273138    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" event={"ID":"0afd87a2-5d84-4854-bc44-6835c5be75cb","Type":"ContainerDied","Data":"decdbd4a0c543a3e716a5ccdc2cc09e41d0118304551c43b19323ebad7024fee"}
Jul 29 10:50:13.352489 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.352462    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:50:13.477862 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.477822    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-rbac-proxy-sar-config\") pod \"0afd87a2-5d84-4854-bc44-6835c5be75cb\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") "
Jul 29 10:50:13.478043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.477893    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0afd87a2-5d84-4854-bc44-6835c5be75cb-proxy-tls\") pod \"0afd87a2-5d84-4854-bc44-6835c5be75cb\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") "
Jul 29 10:50:13.478043 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.477950    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xql2x\" (UniqueName: \"kubernetes.io/projected/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-api-access-xql2x\") pod \"0afd87a2-5d84-4854-bc44-6835c5be75cb\" (UID: \"0afd87a2-5d84-4854-bc44-6835c5be75cb\") "
Jul 29 10:50:13.478342 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.478312    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0afd87a2-5d84-4854-bc44-6835c5be75cb" (UID: "0afd87a2-5d84-4854-bc44-6835c5be75cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:50:13.480222 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.480195    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-api-access-xql2x" (OuterVolumeSpecName: "kube-api-access-xql2x") pod "0afd87a2-5d84-4854-bc44-6835c5be75cb" (UID: "0afd87a2-5d84-4854-bc44-6835c5be75cb"). InnerVolumeSpecName "kube-api-access-xql2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:50:13.480222 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.480197    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0afd87a2-5d84-4854-bc44-6835c5be75cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0afd87a2-5d84-4854-bc44-6835c5be75cb" (UID: "0afd87a2-5d84-4854-bc44-6835c5be75cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:50:13.578548 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.578454    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0afd87a2-5d84-4854-bc44-6835c5be75cb-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:50:13.578548 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.578485    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xql2x\" (UniqueName: \"kubernetes.io/projected/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-api-access-xql2x\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:50:13.578548 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:13.578495    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0afd87a2-5d84-4854-bc44-6835c5be75cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:50:14.279377 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:14.279341    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q" event={"ID":"0afd87a2-5d84-4854-bc44-6835c5be75cb","Type":"ContainerDied","Data":"986e82afd2365d0a842258ac058726736fe84a3308616b73aff8097e1afb48c9"}
Jul 29 10:50:14.279874 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:14.279432    2572 scope.go:117] "RemoveContainer" containerID="951dea11166ca2f30d26454b1066370a4d1d3fde62bcdd9de49e2d4f66bb9fc5"
Jul 29 10:50:14.279874 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:14.279356    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"
Jul 29 10:50:14.289142 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:14.289115    2572 scope.go:117] "RemoveContainer" containerID="decdbd4a0c543a3e716a5ccdc2cc09e41d0118304551c43b19323ebad7024fee"
Jul 29 10:50:14.300463 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:14.300432    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"]
Jul 29 10:50:14.303659 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:14.303633    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6242a-predictor-cbf7db67b-9cr5q"]
Jul 29 10:50:14.310232 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:14.310195    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" path="/var/lib/kubelet/pods/0afd87a2-5d84-4854-bc44-6835c5be75cb/volumes"
Jul 29 10:50:17.227462 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:17.227417    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:50:17.271552 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:17.271520    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:50:17.272095 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:17.272068    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 29 10:50:18.279087 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:18.279056    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:50:22.226970 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:22.226903    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:50:22.227442 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:22.227069    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:27.226531 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:27.226486    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:50:27.272948 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:27.272886    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 29 10:50:32.226727 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:32.226687    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:50:37.226824 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:37.226777    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:50:37.272838 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:37.272794    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 29 10:50:40.044533 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.044502    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:40.074938 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.074888    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-proxy-tls\") pod \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\" (UID: \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\") "
Jul 29 10:50:40.075125 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.074986    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-openshift-service-ca-bundle\") pod \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\" (UID: \"c10b7ba0-85a3-4dd2-a8b6-3c9809084551\") "
Jul 29 10:50:40.075431 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.075400    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c10b7ba0-85a3-4dd2-a8b6-3c9809084551" (UID: "c10b7ba0-85a3-4dd2-a8b6-3c9809084551"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:50:40.077093 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.077065    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c10b7ba0-85a3-4dd2-a8b6-3c9809084551" (UID: "c10b7ba0-85a3-4dd2-a8b6-3c9809084551"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:50:40.176182 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.176087    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:50:40.176182 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.176121    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10b7ba0-85a3-4dd2-a8b6-3c9809084551-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:50:40.387015 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.386980    2572 generic.go:356] "Generic (PLEG): container finished" podID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerID="eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510" exitCode=0
Jul 29 10:50:40.387194 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.387028    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" event={"ID":"c10b7ba0-85a3-4dd2-a8b6-3c9809084551","Type":"ContainerDied","Data":"eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510"}
Jul 29 10:50:40.387194 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.387039    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"
Jul 29 10:50:40.387194 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.387096    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45" event={"ID":"c10b7ba0-85a3-4dd2-a8b6-3c9809084551","Type":"ContainerDied","Data":"867d5c1705ae7a36afc667176034b67757bfffd5cb197fc896933aa88a7042ae"}
Jul 29 10:50:40.387194 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.387118    2572 scope.go:117] "RemoveContainer" containerID="eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510"
Jul 29 10:50:40.395529 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.395504    2572 scope.go:117] "RemoveContainer" containerID="eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510"
Jul 29 10:50:40.395840 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:50:40.395815    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510\": container with ID starting with eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510 not found: ID does not exist" containerID="eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510"
Jul 29 10:50:40.395917 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.395856    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510"} err="failed to get container status \"eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510\": rpc error: code = NotFound desc = could not find container \"eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510\": container with ID starting with eb14c5a3142dfed95098d4966b0ca01818336704e5ea27fa9fffaa92944e5510 not found: ID does not exist"
Jul 29 10:50:40.402159 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.402126    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"]
Jul 29 10:50:40.405805 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:40.405771    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-6242a-6df9fc859f-jpk45"]
Jul 29 10:50:42.310003 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:42.309965    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" path="/var/lib/kubelet/pods/c10b7ba0-85a3-4dd2-a8b6-3c9809084551/volumes"
Jul 29 10:50:47.272228 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:47.272187    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 29 10:50:57.273084 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:50:57.273051    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:51:10.134896 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.134853    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"]
Jul 29 10:51:10.135392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.135217    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kube-rbac-proxy"
Jul 29 10:51:10.135392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.135233    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kube-rbac-proxy"
Jul 29 10:51:10.135392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.135254    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a"
Jul 29 10:51:10.135392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.135263    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a"
Jul 29 10:51:10.135392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.135274    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container"
Jul 29 10:51:10.135392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.135283    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container"
Jul 29 10:51:10.135392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.135352    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kube-rbac-proxy"
Jul 29 10:51:10.135392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.135362    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0afd87a2-5d84-4854-bc44-6835c5be75cb" containerName="kserve-container"
Jul 29 10:51:10.135392 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.135371    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10b7ba0-85a3-4dd2-a8b6-3c9809084551" containerName="sequence-graph-6242a"
Jul 29 10:51:10.138345 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.138323    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:51:10.140151 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.140123    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-0934b-kube-rbac-proxy-sar-config\""
Jul 29 10:51:10.140386 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.140372    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-0934b-serving-cert\""
Jul 29 10:51:10.149077 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.149044    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"]
Jul 29 10:51:10.212623 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.212576    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc81178b-a296-4689-900c-7411ef6a3af8-openshift-service-ca-bundle\") pod \"sequence-graph-0934b-74995788f-j8psd\" (UID: \"bc81178b-a296-4689-900c-7411ef6a3af8\") " pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:51:10.212623 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.212624    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc81178b-a296-4689-900c-7411ef6a3af8-proxy-tls\") pod \"sequence-graph-0934b-74995788f-j8psd\" (UID: \"bc81178b-a296-4689-900c-7411ef6a3af8\") " pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:51:10.313686 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.313651    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc81178b-a296-4689-900c-7411ef6a3af8-openshift-service-ca-bundle\") pod \"sequence-graph-0934b-74995788f-j8psd\" (UID: \"bc81178b-a296-4689-900c-7411ef6a3af8\") " pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:51:10.313686 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.313686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc81178b-a296-4689-900c-7411ef6a3af8-proxy-tls\") pod \"sequence-graph-0934b-74995788f-j8psd\" (UID: \"bc81178b-a296-4689-900c-7411ef6a3af8\") " pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:51:10.314304 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.314273    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc81178b-a296-4689-900c-7411ef6a3af8-openshift-service-ca-bundle\") pod \"sequence-graph-0934b-74995788f-j8psd\" (UID: \"bc81178b-a296-4689-900c-7411ef6a3af8\") " pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:51:10.316147 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.316125    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc81178b-a296-4689-900c-7411ef6a3af8-proxy-tls\") pod \"sequence-graph-0934b-74995788f-j8psd\" (UID: \"bc81178b-a296-4689-900c-7411ef6a3af8\") " pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:51:10.455692 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.455588    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:51:10.595572 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:10.595534    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"]
Jul 29 10:51:10.598836 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:51:10.598799    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc81178b_a296_4689_900c_7411ef6a3af8.slice/crio-b613d0b37a43b129d06a6ae80f6cb4e8eb04f05e8ea1fab69a8537da14415772 WatchSource:0}: Error finding container b613d0b37a43b129d06a6ae80f6cb4e8eb04f05e8ea1fab69a8537da14415772: Status 404 returned error can't find the container with id b613d0b37a43b129d06a6ae80f6cb4e8eb04f05e8ea1fab69a8537da14415772
Jul 29 10:51:11.515999 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:11.515960    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" event={"ID":"bc81178b-a296-4689-900c-7411ef6a3af8","Type":"ContainerStarted","Data":"9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882"}
Jul 29 10:51:11.516496 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:11.516012    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" event={"ID":"bc81178b-a296-4689-900c-7411ef6a3af8","Type":"ContainerStarted","Data":"b613d0b37a43b129d06a6ae80f6cb4e8eb04f05e8ea1fab69a8537da14415772"}
Jul 29 10:51:11.516496 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:11.516043    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:51:11.531833 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:11.531768    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" podStartSLOduration=1.531750992 podStartE2EDuration="1.531750992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:51:11.530618584 +0000 UTC m=+1353.758865134" watchObservedRunningTime="2026-07-29 10:51:11.531750992 +0000 UTC m=+1353.759997542"
Jul 29 10:51:17.525775 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:51:17.525738    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:58:24.853289 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:24.853253    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"]
Jul 29 10:58:24.855773 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:24.853477    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23" containerID="cri-o://97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea" gracePeriod=30
Jul 29 10:58:25.109397 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.109315    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"]
Jul 29 10:58:25.112977 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.112954    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"]
Jul 29 10:58:25.113089 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.113031    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.113381 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.113149    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container" containerID="cri-o://768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70" gracePeriod=30
Jul 29 10:58:25.113381 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.113230    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kube-rbac-proxy" containerID="cri-o://8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148" gracePeriod=30
Jul 29 10:58:25.115190 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.114996    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f5066-kube-rbac-proxy-sar-config\""
Jul 29 10:58:25.115190 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.115017    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f5066-predictor-serving-cert\""
Jul 29 10:58:25.123396 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.123370    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"]
Jul 29 10:58:25.240398 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.240347    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/159f0b39-a911-48cc-b0a2-8034742e990a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f5066-predictor-569b8fdd8-tgl8h\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") " pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.240577 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.240412    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jtd\" (UniqueName: \"kubernetes.io/projected/159f0b39-a911-48cc-b0a2-8034742e990a-kube-api-access-d2jtd\") pod \"success-200-isvc-f5066-predictor-569b8fdd8-tgl8h\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") " pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.240577 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.240465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/159f0b39-a911-48cc-b0a2-8034742e990a-proxy-tls\") pod \"success-200-isvc-f5066-predictor-569b8fdd8-tgl8h\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") " pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.341892 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.341849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/159f0b39-a911-48cc-b0a2-8034742e990a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f5066-predictor-569b8fdd8-tgl8h\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") " pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.342125 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.341902    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jtd\" (UniqueName: \"kubernetes.io/projected/159f0b39-a911-48cc-b0a2-8034742e990a-kube-api-access-d2jtd\") pod \"success-200-isvc-f5066-predictor-569b8fdd8-tgl8h\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") " pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.342125 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.341958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/159f0b39-a911-48cc-b0a2-8034742e990a-proxy-tls\") pod \"success-200-isvc-f5066-predictor-569b8fdd8-tgl8h\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") " pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.342518 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.342496    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/159f0b39-a911-48cc-b0a2-8034742e990a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f5066-predictor-569b8fdd8-tgl8h\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") " pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.344462 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.344442    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/159f0b39-a911-48cc-b0a2-8034742e990a-proxy-tls\") pod \"success-200-isvc-f5066-predictor-569b8fdd8-tgl8h\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") " pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.349751 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.349721    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jtd\" (UniqueName: \"kubernetes.io/projected/159f0b39-a911-48cc-b0a2-8034742e990a-kube-api-access-d2jtd\") pod \"success-200-isvc-f5066-predictor-569b8fdd8-tgl8h\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") " pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.432568 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.432529    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:25.570694 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:25.570657    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"]
Jul 29 10:58:25.574387 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:58:25.574358    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159f0b39_a911_48cc_b0a2_8034742e990a.slice/crio-4c11fc10d9ec1fc048e3669c3f115ae03fca336c32a7551c8938ddbefcfafc8b WatchSource:0}: Error finding container 4c11fc10d9ec1fc048e3669c3f115ae03fca336c32a7551c8938ddbefcfafc8b: Status 404 returned error can't find the container with id 4c11fc10d9ec1fc048e3669c3f115ae03fca336c32a7551c8938ddbefcfafc8b
Jul 29 10:58:26.136567 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:26.136519    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" event={"ID":"159f0b39-a911-48cc-b0a2-8034742e990a","Type":"ContainerStarted","Data":"dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5"}
Jul 29 10:58:26.137031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:26.136593    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" event={"ID":"159f0b39-a911-48cc-b0a2-8034742e990a","Type":"ContainerStarted","Data":"a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74"}
Jul 29 10:58:26.137031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:26.136621    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" event={"ID":"159f0b39-a911-48cc-b0a2-8034742e990a","Type":"ContainerStarted","Data":"4c11fc10d9ec1fc048e3669c3f115ae03fca336c32a7551c8938ddbefcfafc8b"}
Jul 29 10:58:26.137031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:26.136657    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:26.137031 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:26.136687    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:26.138367 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:26.138336    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 29 10:58:26.138909 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:26.138889    2572 generic.go:356] "Generic (PLEG): container finished" podID="44d53f76-1981-4b72-b824-2c949abdf491" containerID="8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148" exitCode=2
Jul 29 10:58:26.139012 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:26.138966    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" event={"ID":"44d53f76-1981-4b72-b824-2c949abdf491","Type":"ContainerDied","Data":"8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148"}
Jul 29 10:58:26.153166 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:26.153106    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podStartSLOduration=1.1530856090000001 podStartE2EDuration="1.153085609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:58:26.152675503 +0000 UTC m=+1788.380922089" watchObservedRunningTime="2026-07-29 10:58:26.153085609 +0000 UTC m=+1788.381332160"
Jul 29 10:58:27.143999 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:27.143957    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 29 10:58:28.029046 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.029001    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 29 10:58:28.033330 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.033302    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 29 10:58:28.279200 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.279106    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:58:28.370256 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.370230    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:58:28.468199 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.468164    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d53f76-1981-4b72-b824-2c949abdf491-proxy-tls\") pod \"44d53f76-1981-4b72-b824-2c949abdf491\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") "
Jul 29 10:58:28.468379 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.468237    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmgrs\" (UniqueName: \"kubernetes.io/projected/44d53f76-1981-4b72-b824-2c949abdf491-kube-api-access-nmgrs\") pod \"44d53f76-1981-4b72-b824-2c949abdf491\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") "
Jul 29 10:58:28.468379 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.468285    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d53f76-1981-4b72-b824-2c949abdf491-kube-rbac-proxy-sar-config\") pod \"44d53f76-1981-4b72-b824-2c949abdf491\" (UID: \"44d53f76-1981-4b72-b824-2c949abdf491\") "
Jul 29 10:58:28.468679 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.468651    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d53f76-1981-4b72-b824-2c949abdf491-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44d53f76-1981-4b72-b824-2c949abdf491" (UID: "44d53f76-1981-4b72-b824-2c949abdf491"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:58:28.470401 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.470380    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d53f76-1981-4b72-b824-2c949abdf491-kube-api-access-nmgrs" (OuterVolumeSpecName: "kube-api-access-nmgrs") pod "44d53f76-1981-4b72-b824-2c949abdf491" (UID: "44d53f76-1981-4b72-b824-2c949abdf491"). InnerVolumeSpecName "kube-api-access-nmgrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:58:28.470486 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.470425    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d53f76-1981-4b72-b824-2c949abdf491-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44d53f76-1981-4b72-b824-2c949abdf491" (UID: "44d53f76-1981-4b72-b824-2c949abdf491"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:58:28.569114 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.569022    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d53f76-1981-4b72-b824-2c949abdf491-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:58:28.569114 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.569053    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d53f76-1981-4b72-b824-2c949abdf491-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:58:28.569114 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:28.569063    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmgrs\" (UniqueName: \"kubernetes.io/projected/44d53f76-1981-4b72-b824-2c949abdf491-kube-api-access-nmgrs\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:58:29.155832 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.155793    2572 generic.go:356] "Generic (PLEG): container finished" podID="44d53f76-1981-4b72-b824-2c949abdf491" containerID="768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70" exitCode=0
Jul 29 10:58:29.156051 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.155835    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" event={"ID":"44d53f76-1981-4b72-b824-2c949abdf491","Type":"ContainerDied","Data":"768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70"}
Jul 29 10:58:29.156051 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.155858    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"
Jul 29 10:58:29.156051 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.155917    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s" event={"ID":"44d53f76-1981-4b72-b824-2c949abdf491","Type":"ContainerDied","Data":"aed62492e1b313a189ce30742b39ccb839623b383fd43976ff5e24a803965e78"}
Jul 29 10:58:29.156051 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.155976    2572 scope.go:117] "RemoveContainer" containerID="8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148"
Jul 29 10:58:29.164932 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.164903    2572 scope.go:117] "RemoveContainer" containerID="768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70"
Jul 29 10:58:29.173061 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.173043    2572 scope.go:117] "RemoveContainer" containerID="8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148"
Jul 29 10:58:29.173383 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:58:29.173361    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148\": container with ID starting with 8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148 not found: ID does not exist" containerID="8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148"
Jul 29 10:58:29.173455 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.173397    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148"} err="failed to get container status \"8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148\": rpc error: code = NotFound desc = could not find container \"8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148\": container with ID starting with 8dfed3cc91dff515d1715c692d5784489860cdb9cf5d79eeaa52bc9aefcae148 not found: ID does not exist"
Jul 29 10:58:29.173455 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.173422    2572 scope.go:117] "RemoveContainer" containerID="768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70"
Jul 29 10:58:29.173657 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:58:29.173639    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70\": container with ID starting with 768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70 not found: ID does not exist" containerID="768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70"
Jul 29 10:58:29.173697 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.173665    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70"} err="failed to get container status \"768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70\": rpc error: code = NotFound desc = could not find container \"768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70\": container with ID starting with 768e243aaf89c95622a5dfeba9332291f8c9f7950eb968bb2dbcb7d8bfc8ad70 not found: ID does not exist"
Jul 29 10:58:29.176303 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.176279    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"]
Jul 29 10:58:29.181912 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:29.181887    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2d23-predictor-77b7bd766d-vx77s"]
Jul 29 10:58:30.309278 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:30.309199    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d53f76-1981-4b72-b824-2c949abdf491" path="/var/lib/kubelet/pods/44d53f76-1981-4b72-b824-2c949abdf491/volumes"
Jul 29 10:58:32.148678 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:32.148648    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:58:32.149188 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:32.149155    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 29 10:58:33.277942 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:33.277890    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:58:38.277292 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:38.277243    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:58:38.277778 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:38.277384    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:58:42.149571 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:42.149525    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 29 10:58:43.277424 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:43.277374    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:58:48.278175 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:48.278130    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:58:52.149716 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:52.149672    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 29 10:58:53.277742 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:53.277694    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:58:55.012299 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.012266    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:58:55.077737 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.077697    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-openshift-service-ca-bundle\") pod \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\" (UID: \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\") "
Jul 29 10:58:55.077897 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.077774    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-proxy-tls\") pod \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\" (UID: \"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12\") "
Jul 29 10:58:55.078155 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.078130    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" (UID: "f8bc01a7-7322-4593-92ff-5f0e5e9a4e12"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:58:55.079996 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.079971    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" (UID: "f8bc01a7-7322-4593-92ff-5f0e5e9a4e12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:58:55.178616 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.178580    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:58:55.178616 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.178612    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:58:55.257963 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.257912    2572 generic.go:356] "Generic (PLEG): container finished" podID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerID="97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea" exitCode=0
Jul 29 10:58:55.258152 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.257975    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"
Jul 29 10:58:55.258152 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.257994    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" event={"ID":"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12","Type":"ContainerDied","Data":"97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea"}
Jul 29 10:58:55.258152 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.258053    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2" event={"ID":"f8bc01a7-7322-4593-92ff-5f0e5e9a4e12","Type":"ContainerDied","Data":"ead4b7f2ddbb2e37915c7660ab13e1c9b61e04a107b2b686953382aa0a676db6"}
Jul 29 10:58:55.258152 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.258078    2572 scope.go:117] "RemoveContainer" containerID="97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea"
Jul 29 10:58:55.267061 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.267038    2572 scope.go:117] "RemoveContainer" containerID="97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea"
Jul 29 10:58:55.267400 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:58:55.267380    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea\": container with ID starting with 97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea not found: ID does not exist" containerID="97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea"
Jul 29 10:58:55.267442 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.267412    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea"} err="failed to get container status \"97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea\": rpc error: code = NotFound desc = could not find container \"97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea\": container with ID starting with 97e8d60a3f5c70b521a44f30086217d050e6fb937e55dea3ec35cb4ea8f9c8ea not found: ID does not exist"
Jul 29 10:58:55.277724 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.277694    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"]
Jul 29 10:58:55.280934 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:55.280892    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d2d23-557d84f99-bwkd2"]
Jul 29 10:58:56.309243 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:58:56.309210    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" path="/var/lib/kubelet/pods/f8bc01a7-7322-4593-92ff-5f0e5e9a4e12/volumes"
Jul 29 10:59:02.149585 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:02.149539    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 29 10:59:12.150685 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:12.150651    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:59:24.758205 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.758166    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"]
Jul 29 10:59:24.758774 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.758438    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b" containerID="cri-o://9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882" gracePeriod=30
Jul 29 10:59:24.883516 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.883471    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"]
Jul 29 10:59:24.883833 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.883802    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container" containerID="cri-o://0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7" gracePeriod=30
Jul 29 10:59:24.883980 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.883902    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kube-rbac-proxy" containerID="cri-o://a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9" gracePeriod=30
Jul 29 10:59:24.949996 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.949956    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"]
Jul 29 10:59:24.950270 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.950257    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kube-rbac-proxy"
Jul 29 10:59:24.950270 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.950271    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kube-rbac-proxy"
Jul 29 10:59:24.950378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.950283    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container"
Jul 29 10:59:24.950378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.950289    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container"
Jul 29 10:59:24.950378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.950299    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23"
Jul 29 10:59:24.950378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.950305    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23"
Jul 29 10:59:24.950378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.950355    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8bc01a7-7322-4593-92ff-5f0e5e9a4e12" containerName="ensemble-graph-d2d23"
Jul 29 10:59:24.950378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.950362    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kube-rbac-proxy"
Jul 29 10:59:24.950378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.950368    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d53f76-1981-4b72-b824-2c949abdf491" containerName="kserve-container"
Jul 29 10:59:24.953555 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.953526    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:24.955310 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.955288    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-965a2-predictor-serving-cert\""
Jul 29 10:59:24.955417 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.955370    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-965a2-kube-rbac-proxy-sar-config\""
Jul 29 10:59:24.964670 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:24.964639    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"]
Jul 29 10:59:25.090832 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.090744    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"]
Jul 29 10:59:25.094343 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.094317    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:25.096346 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.096322    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f5066-kube-rbac-proxy-sar-config\""
Jul 29 10:59:25.096469 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.096322    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f5066-serving-cert\""
Jul 29 10:59:25.106343 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.106314    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"]
Jul 29 10:59:25.115185 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.115144    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt6cv\" (UniqueName: \"kubernetes.io/projected/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-api-access-jt6cv\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.115356 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.115204    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.115356 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.115276    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d2a27d-baba-45b7-a758-cbe3a4ee264c-proxy-tls\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.216102 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.216059    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c16f1604-9687-4840-9dd2-965b42fc9e7f-openshift-service-ca-bundle\") pod \"splitter-graph-f5066-5dd8b587f8-6xmdh\" (UID: \"c16f1604-9687-4840-9dd2-965b42fc9e7f\") " pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:25.216304 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.216114    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt6cv\" (UniqueName: \"kubernetes.io/projected/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-api-access-jt6cv\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.216304 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.216171    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.216304 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.216266    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d2a27d-baba-45b7-a758-cbe3a4ee264c-proxy-tls\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.216468 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:59:25.216393    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-965a2-predictor-serving-cert: secret "success-200-isvc-965a2-predictor-serving-cert" not found
Jul 29 10:59:25.216468 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:59:25.216456    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43d2a27d-baba-45b7-a758-cbe3a4ee264c-proxy-tls podName:43d2a27d-baba-45b7-a758-cbe3a4ee264c nodeName:}" failed. No retries permitted until 2026-07-29 10:59:25.716436804 +0000 UTC m=+1847.944683341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/43d2a27d-baba-45b7-a758-cbe3a4ee264c-proxy-tls") pod "success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" (UID: "43d2a27d-baba-45b7-a758-cbe3a4ee264c") : secret "success-200-isvc-965a2-predictor-serving-cert" not found
Jul 29 10:59:25.216698 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.216297    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c16f1604-9687-4840-9dd2-965b42fc9e7f-proxy-tls\") pod \"splitter-graph-f5066-5dd8b587f8-6xmdh\" (UID: \"c16f1604-9687-4840-9dd2-965b42fc9e7f\") " pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:25.216938 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.216897    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.223949 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.223911    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt6cv\" (UniqueName: \"kubernetes.io/projected/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-api-access-jt6cv\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.317428 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.317388    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c16f1604-9687-4840-9dd2-965b42fc9e7f-proxy-tls\") pod \"splitter-graph-f5066-5dd8b587f8-6xmdh\" (UID: \"c16f1604-9687-4840-9dd2-965b42fc9e7f\") " pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:25.317624 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.317533    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c16f1604-9687-4840-9dd2-965b42fc9e7f-openshift-service-ca-bundle\") pod \"splitter-graph-f5066-5dd8b587f8-6xmdh\" (UID: \"c16f1604-9687-4840-9dd2-965b42fc9e7f\") " pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:25.318216 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.318197    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c16f1604-9687-4840-9dd2-965b42fc9e7f-openshift-service-ca-bundle\") pod \"splitter-graph-f5066-5dd8b587f8-6xmdh\" (UID: \"c16f1604-9687-4840-9dd2-965b42fc9e7f\") " pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:25.319977 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.319959    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c16f1604-9687-4840-9dd2-965b42fc9e7f-proxy-tls\") pod \"splitter-graph-f5066-5dd8b587f8-6xmdh\" (UID: \"c16f1604-9687-4840-9dd2-965b42fc9e7f\") " pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:25.376525 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.376490    2572 generic.go:356] "Generic (PLEG): container finished" podID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerID="a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9" exitCode=2
Jul 29 10:59:25.376710 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.376562    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" event={"ID":"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd","Type":"ContainerDied","Data":"a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9"}
Jul 29 10:59:25.405747 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.405708    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:25.540977 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.540832    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"]
Jul 29 10:59:25.543604 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:59:25.543574    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16f1604_9687_4840_9dd2_965b42fc9e7f.slice/crio-aa52567b9bdeb6ccee1f39e7fefc0fc00675e39e94a5192c169eaebf5920ecd6 WatchSource:0}: Error finding container aa52567b9bdeb6ccee1f39e7fefc0fc00675e39e94a5192c169eaebf5920ecd6: Status 404 returned error can't find the container with id aa52567b9bdeb6ccee1f39e7fefc0fc00675e39e94a5192c169eaebf5920ecd6
Jul 29 10:59:25.721208 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.721101    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d2a27d-baba-45b7-a758-cbe3a4ee264c-proxy-tls\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.723626 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.723597    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d2a27d-baba-45b7-a758-cbe3a4ee264c-proxy-tls\") pod \"success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") " pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:25.865047 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:25.865004    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:26.005869 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.005788    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"]
Jul 29 10:59:26.009800 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:59:26.009767    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d2a27d_baba_45b7_a758_cbe3a4ee264c.slice/crio-c01c2dbcd7788f4c7eeef86835be69208f01fed75752757b9ec0e95bf2394ecc WatchSource:0}: Error finding container c01c2dbcd7788f4c7eeef86835be69208f01fed75752757b9ec0e95bf2394ecc: Status 404 returned error can't find the container with id c01c2dbcd7788f4c7eeef86835be69208f01fed75752757b9ec0e95bf2394ecc
Jul 29 10:59:26.382048 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.381991    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" event={"ID":"43d2a27d-baba-45b7-a758-cbe3a4ee264c","Type":"ContainerStarted","Data":"09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7"}
Jul 29 10:59:26.382272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.382080    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:26.382272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.382111    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" event={"ID":"43d2a27d-baba-45b7-a758-cbe3a4ee264c","Type":"ContainerStarted","Data":"f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007"}
Jul 29 10:59:26.382272 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.382144    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" event={"ID":"43d2a27d-baba-45b7-a758-cbe3a4ee264c","Type":"ContainerStarted","Data":"c01c2dbcd7788f4c7eeef86835be69208f01fed75752757b9ec0e95bf2394ecc"}
Jul 29 10:59:26.383457 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.383433    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" event={"ID":"c16f1604-9687-4840-9dd2-965b42fc9e7f","Type":"ContainerStarted","Data":"b578fead8244de118c63f0c9894cc7b8fa71999df2cd4f806eb79dc25e6403a5"}
Jul 29 10:59:26.383579 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.383473    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" event={"ID":"c16f1604-9687-4840-9dd2-965b42fc9e7f","Type":"ContainerStarted","Data":"aa52567b9bdeb6ccee1f39e7fefc0fc00675e39e94a5192c169eaebf5920ecd6"}
Jul 29 10:59:26.383579 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.383559    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:26.400117 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.400061    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" podStartSLOduration=2.400041511 podStartE2EDuration="2.400041511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:59:26.398143645 +0000 UTC m=+1848.626390204" watchObservedRunningTime="2026-07-29 10:59:26.400041511 +0000 UTC m=+1848.628288088"
Jul 29 10:59:26.412800 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:26.412741    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" podStartSLOduration=1.412721155 podStartE2EDuration="1.412721155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:59:26.411572883 +0000 UTC m=+1848.639819433" watchObservedRunningTime="2026-07-29 10:59:26.412721155 +0000 UTC m=+1848.640967707"
Jul 29 10:59:27.268114 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:27.268069    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 29 10:59:27.272487 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:27.272454    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 29 10:59:27.388086 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:27.388047    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:27.389326 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:27.389298    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 29 10:59:27.523616 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:27.523517    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:28.332027 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.331999    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:59:28.393180 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.393144    2572 generic.go:356] "Generic (PLEG): container finished" podID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerID="0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7" exitCode=0
Jul 29 10:59:28.393368 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.393198    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"
Jul 29 10:59:28.393368 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.393205    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" event={"ID":"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd","Type":"ContainerDied","Data":"0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7"}
Jul 29 10:59:28.393368 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.393264    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2" event={"ID":"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd","Type":"ContainerDied","Data":"127fc2a1bd055e2ff5d1eeaa8edc308295bae3261703dd08fac82dcd1d37742f"}
Jul 29 10:59:28.393368 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.393291    2572 scope.go:117] "RemoveContainer" containerID="a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9"
Jul 29 10:59:28.393750 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.393723    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 29 10:59:28.401517 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.401495    2572 scope.go:117] "RemoveContainer" containerID="0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7"
Jul 29 10:59:28.408839 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.408817    2572 scope.go:117] "RemoveContainer" containerID="a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9"
Jul 29 10:59:28.409150 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:59:28.409124    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9\": container with ID starting with a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9 not found: ID does not exist" containerID="a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9"
Jul 29 10:59:28.409216 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.409163    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9"} err="failed to get container status \"a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9\": rpc error: code = NotFound desc = could not find container \"a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9\": container with ID starting with a630be0bc5220d2e3f382451bc6195a8381de805c0ce051e1250e1ea7f18d3a9 not found: ID does not exist"
Jul 29 10:59:28.409216 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.409188    2572 scope.go:117] "RemoveContainer" containerID="0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7"
Jul 29 10:59:28.409506 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:59:28.409482    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7\": container with ID starting with 0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7 not found: ID does not exist" containerID="0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7"
Jul 29 10:59:28.409586 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.409514    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7"} err="failed to get container status \"0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7\": rpc error: code = NotFound desc = could not find container \"0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7\": container with ID starting with 0e15e8885cdc9bc54c9a9bc6d83888161cd4cffc7aba8d01e7e3458a22d537d7 not found: ID does not exist"
Jul 29 10:59:28.445508 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.445473    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckw8m\" (UniqueName: \"kubernetes.io/projected/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-api-access-ckw8m\") pod \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") "
Jul 29 10:59:28.445679 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.445541    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-rbac-proxy-sar-config\") pod \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") "
Jul 29 10:59:28.445679 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.445583    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-proxy-tls\") pod \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\" (UID: \"cded21d0-08a5-4c0d-916b-b27ccc2d6ccd\") "
Jul 29 10:59:28.445940 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.445893    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" (UID: "cded21d0-08a5-4c0d-916b-b27ccc2d6ccd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:59:28.447740 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.447718    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-api-access-ckw8m" (OuterVolumeSpecName: "kube-api-access-ckw8m") pod "cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" (UID: "cded21d0-08a5-4c0d-916b-b27ccc2d6ccd"). InnerVolumeSpecName "kube-api-access-ckw8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:59:28.447828 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.447813    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" (UID: "cded21d0-08a5-4c0d-916b-b27ccc2d6ccd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:59:28.546627 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.546581    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:59:28.546627 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.546622    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:59:28.546627 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.546635    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckw8m\" (UniqueName: \"kubernetes.io/projected/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd-kube-api-access-ckw8m\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:59:28.714064 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.714026    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"]
Jul 29 10:59:28.718353 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:28.718323    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0934b-predictor-5cb5448f5d-ftmm2"]
Jul 29 10:59:30.313139 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:30.311007    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" path="/var/lib/kubelet/pods/cded21d0-08a5-4c0d-916b-b27ccc2d6ccd/volumes"
Jul 29 10:59:32.394203 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:32.394172    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:32.523083 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:32.523040    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:33.398701 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:33.398672    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 10:59:33.399205 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:33.399116    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 29 10:59:35.181149 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.181091    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"]
Jul 29 10:59:35.181524 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.181314    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066" containerID="cri-o://b578fead8244de118c63f0c9894cc7b8fa71999df2cd4f806eb79dc25e6403a5" gracePeriod=30
Jul 29 10:59:35.323061 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.323027    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"]
Jul 29 10:59:35.323343 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.323315    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container" containerID="cri-o://a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74" gracePeriod=30
Jul 29 10:59:35.323507 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.323344    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kube-rbac-proxy" containerID="cri-o://dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5" gracePeriod=30
Jul 29 10:59:35.352454 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.352416    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"]
Jul 29 10:59:35.352731 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.352717    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kube-rbac-proxy"
Jul 29 10:59:35.352731 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.352731    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kube-rbac-proxy"
Jul 29 10:59:35.352845 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.352740    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container"
Jul 29 10:59:35.352845 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.352746    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container"
Jul 29 10:59:35.352845 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.352791    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kube-rbac-proxy"
Jul 29 10:59:35.352845 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.352799    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cded21d0-08a5-4c0d-916b-b27ccc2d6ccd" containerName="kserve-container"
Jul 29 10:59:35.357403 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.357377    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:35.359275 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.359244    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-afa2d-kube-rbac-proxy-sar-config\""
Jul 29 10:59:35.359410 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.359383    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-afa2d-predictor-serving-cert\""
Jul 29 10:59:35.366835 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.366806    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"]
Jul 29 10:59:35.498200 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.498166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dea421-e029-486a-a81b-cc4e6f1b0a60-proxy-tls\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:35.498378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.498245    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:35.498378 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.498317    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrbf\" (UniqueName: \"kubernetes.io/projected/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-api-access-6nrbf\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:35.598858 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.598802    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:35.599102 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.598902    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrbf\" (UniqueName: \"kubernetes.io/projected/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-api-access-6nrbf\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:35.599102 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.598943    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dea421-e029-486a-a81b-cc4e6f1b0a60-proxy-tls\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:35.599102 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:59:35.599047    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-serving-cert: secret "success-200-isvc-afa2d-predictor-serving-cert" not found
Jul 29 10:59:35.599278 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:59:35.599108    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91dea421-e029-486a-a81b-cc4e6f1b0a60-proxy-tls podName:91dea421-e029-486a-a81b-cc4e6f1b0a60 nodeName:}" failed. No retries permitted until 2026-07-29 10:59:36.099089383 +0000 UTC m=+1858.327335911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/91dea421-e029-486a-a81b-cc4e6f1b0a60-proxy-tls") pod "success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" (UID: "91dea421-e029-486a-a81b-cc4e6f1b0a60") : secret "success-200-isvc-afa2d-predictor-serving-cert" not found
Jul 29 10:59:35.599584 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.599556    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:35.606837 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:35.606800    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrbf\" (UniqueName: \"kubernetes.io/projected/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-api-access-6nrbf\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:36.102113 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:36.102072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dea421-e029-486a-a81b-cc4e6f1b0a60-proxy-tls\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:36.104604 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:36.104575    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dea421-e029-486a-a81b-cc4e6f1b0a60-proxy-tls\") pod \"success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") " pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:36.269433 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:36.269385    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:36.418466 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:36.418433    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"]
Jul 29 10:59:36.421582 ip-10-0-129-138 kubenswrapper[2572]: W0729 10:59:36.421540    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91dea421_e029_486a_a81b_cc4e6f1b0a60.slice/crio-866318d660a0d75f657a7d2617a5e3c768c776912087da7555b8ff77b159de66 WatchSource:0}: Error finding container 866318d660a0d75f657a7d2617a5e3c768c776912087da7555b8ff77b159de66: Status 404 returned error can't find the container with id 866318d660a0d75f657a7d2617a5e3c768c776912087da7555b8ff77b159de66
Jul 29 10:59:36.424501 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:36.424468    2572 generic.go:356] "Generic (PLEG): container finished" podID="159f0b39-a911-48cc-b0a2-8034742e990a" containerID="dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5" exitCode=2
Jul 29 10:59:36.424590 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:36.424543    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" event={"ID":"159f0b39-a911-48cc-b0a2-8034742e990a","Type":"ContainerDied","Data":"dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5"}
Jul 29 10:59:37.144999 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.144949    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 29 10:59:37.392040 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.391994    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:37.430469 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.430372    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" event={"ID":"91dea421-e029-486a-a81b-cc4e6f1b0a60","Type":"ContainerStarted","Data":"15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2"}
Jul 29 10:59:37.430469 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.430463    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" event={"ID":"91dea421-e029-486a-a81b-cc4e6f1b0a60","Type":"ContainerStarted","Data":"a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21"}
Jul 29 10:59:37.430685 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.430492    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" event={"ID":"91dea421-e029-486a-a81b-cc4e6f1b0a60","Type":"ContainerStarted","Data":"866318d660a0d75f657a7d2617a5e3c768c776912087da7555b8ff77b159de66"}
Jul 29 10:59:37.430685 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.430515    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:37.430685 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.430540    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:37.431899 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.431869    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 29 10:59:37.448056 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.447990    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" podStartSLOduration=2.44796544 podStartE2EDuration="2.44796544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:59:37.446420958 +0000 UTC m=+1859.674667519" watchObservedRunningTime="2026-07-29 10:59:37.44796544 +0000 UTC m=+1859.676211991"
Jul 29 10:59:37.523124 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.523085    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:37.523303 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:37.523214    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:59:38.434541 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.434494    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 29 10:59:38.683497 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.683472    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:59:38.828634 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.828583    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/159f0b39-a911-48cc-b0a2-8034742e990a-kube-rbac-proxy-sar-config\") pod \"159f0b39-a911-48cc-b0a2-8034742e990a\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") "
Jul 29 10:59:38.828634 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.828637    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/159f0b39-a911-48cc-b0a2-8034742e990a-proxy-tls\") pod \"159f0b39-a911-48cc-b0a2-8034742e990a\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") "
Jul 29 10:59:38.828905 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.828679    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2jtd\" (UniqueName: \"kubernetes.io/projected/159f0b39-a911-48cc-b0a2-8034742e990a-kube-api-access-d2jtd\") pod \"159f0b39-a911-48cc-b0a2-8034742e990a\" (UID: \"159f0b39-a911-48cc-b0a2-8034742e990a\") "
Jul 29 10:59:38.829056 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.829032    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159f0b39-a911-48cc-b0a2-8034742e990a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "159f0b39-a911-48cc-b0a2-8034742e990a" (UID: "159f0b39-a911-48cc-b0a2-8034742e990a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:59:38.830879 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.830849    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159f0b39-a911-48cc-b0a2-8034742e990a-kube-api-access-d2jtd" (OuterVolumeSpecName: "kube-api-access-d2jtd") pod "159f0b39-a911-48cc-b0a2-8034742e990a" (UID: "159f0b39-a911-48cc-b0a2-8034742e990a"). InnerVolumeSpecName "kube-api-access-d2jtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:59:38.830961 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.830849    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/159f0b39-a911-48cc-b0a2-8034742e990a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "159f0b39-a911-48cc-b0a2-8034742e990a" (UID: "159f0b39-a911-48cc-b0a2-8034742e990a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:59:38.929523 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.929414    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/159f0b39-a911-48cc-b0a2-8034742e990a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:59:38.929523 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.929461    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/159f0b39-a911-48cc-b0a2-8034742e990a-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:59:38.929523 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:38.929471    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2jtd\" (UniqueName: \"kubernetes.io/projected/159f0b39-a911-48cc-b0a2-8034742e990a-kube-api-access-d2jtd\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:59:39.439622 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.439588    2572 generic.go:356] "Generic (PLEG): container finished" podID="159f0b39-a911-48cc-b0a2-8034742e990a" containerID="a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74" exitCode=0
Jul 29 10:59:39.440080 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.439648    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"
Jul 29 10:59:39.440080 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.439677    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" event={"ID":"159f0b39-a911-48cc-b0a2-8034742e990a","Type":"ContainerDied","Data":"a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74"}
Jul 29 10:59:39.440080 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.439739    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h" event={"ID":"159f0b39-a911-48cc-b0a2-8034742e990a","Type":"ContainerDied","Data":"4c11fc10d9ec1fc048e3669c3f115ae03fca336c32a7551c8938ddbefcfafc8b"}
Jul 29 10:59:39.440080 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.439766    2572 scope.go:117] "RemoveContainer" containerID="dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5"
Jul 29 10:59:39.448529 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.448508    2572 scope.go:117] "RemoveContainer" containerID="a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74"
Jul 29 10:59:39.456100 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.456074    2572 scope.go:117] "RemoveContainer" containerID="dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5"
Jul 29 10:59:39.456374 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:59:39.456354    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5\": container with ID starting with dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5 not found: ID does not exist" containerID="dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5"
Jul 29 10:59:39.456415 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.456385    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5"} err="failed to get container status \"dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5\": rpc error: code = NotFound desc = could not find container \"dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5\": container with ID starting with dee7cdd26989f44d4f8f2cd2e65842402b14ed072ae9150dc88d8a610c6927f5 not found: ID does not exist"
Jul 29 10:59:39.456415 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.456405    2572 scope.go:117] "RemoveContainer" containerID="a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74"
Jul 29 10:59:39.456669 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:59:39.456654    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74\": container with ID starting with a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74 not found: ID does not exist" containerID="a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74"
Jul 29 10:59:39.456732 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.456672    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74"} err="failed to get container status \"a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74\": rpc error: code = NotFound desc = could not find container \"a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74\": container with ID starting with a18184ed58d6c3c80fec0dbcbbdc9363f4b676ac59d734d2129c65a23795cd74 not found: ID does not exist"
Jul 29 10:59:39.460849 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.460825    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"]
Jul 29 10:59:39.464308 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:39.464286    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f5066-predictor-569b8fdd8-tgl8h"]
Jul 29 10:59:40.310677 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:40.310634    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" path="/var/lib/kubelet/pods/159f0b39-a911-48cc-b0a2-8034742e990a/volumes"
Jul 29 10:59:42.391868 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:42.391830    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:42.523716 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:42.523675    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:43.400128 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:43.400080    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 29 10:59:43.439663 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:43.439623    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 10:59:43.440218 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:43.440192    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 29 10:59:47.391980 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:47.391912    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:47.392556 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:47.392072    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 10:59:47.523193 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:47.523150    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:52.392218 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:52.392169    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:52.523829 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:52.523782    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:59:53.400072 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:53.400028    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 29 10:59:53.440321 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:53.440282    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 29 10:59:54.951147 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:54.951117    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:59:55.054206 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.054095    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc81178b-a296-4689-900c-7411ef6a3af8-openshift-service-ca-bundle\") pod \"bc81178b-a296-4689-900c-7411ef6a3af8\" (UID: \"bc81178b-a296-4689-900c-7411ef6a3af8\") "
Jul 29 10:59:55.054206 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.054209    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc81178b-a296-4689-900c-7411ef6a3af8-proxy-tls\") pod \"bc81178b-a296-4689-900c-7411ef6a3af8\" (UID: \"bc81178b-a296-4689-900c-7411ef6a3af8\") "
Jul 29 10:59:55.054584 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.054546    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc81178b-a296-4689-900c-7411ef6a3af8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bc81178b-a296-4689-900c-7411ef6a3af8" (UID: "bc81178b-a296-4689-900c-7411ef6a3af8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:59:55.056352 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.056326    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc81178b-a296-4689-900c-7411ef6a3af8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc81178b-a296-4689-900c-7411ef6a3af8" (UID: "bc81178b-a296-4689-900c-7411ef6a3af8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:59:55.155219 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.155180    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc81178b-a296-4689-900c-7411ef6a3af8-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:59:55.155219 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.155213    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc81178b-a296-4689-900c-7411ef6a3af8-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 10:59:55.502546 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.502509    2572 generic.go:356] "Generic (PLEG): container finished" podID="bc81178b-a296-4689-900c-7411ef6a3af8" containerID="9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882" exitCode=0
Jul 29 10:59:55.502705 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.502552    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"
Jul 29 10:59:55.502705 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.502592    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" event={"ID":"bc81178b-a296-4689-900c-7411ef6a3af8","Type":"ContainerDied","Data":"9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882"}
Jul 29 10:59:55.502705 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.502643    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd" event={"ID":"bc81178b-a296-4689-900c-7411ef6a3af8","Type":"ContainerDied","Data":"b613d0b37a43b129d06a6ae80f6cb4e8eb04f05e8ea1fab69a8537da14415772"}
Jul 29 10:59:55.502705 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.502665    2572 scope.go:117] "RemoveContainer" containerID="9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882"
Jul 29 10:59:55.511188 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.511167    2572 scope.go:117] "RemoveContainer" containerID="9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882"
Jul 29 10:59:55.511488 ip-10-0-129-138 kubenswrapper[2572]: E0729 10:59:55.511468    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882\": container with ID starting with 9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882 not found: ID does not exist" containerID="9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882"
Jul 29 10:59:55.511554 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.511496    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882"} err="failed to get container status \"9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882\": rpc error: code = NotFound desc = could not find container \"9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882\": container with ID starting with 9cf6a76c4a6bbe35075a95591da0c97ebacdde6713709bbc8b943b6b49133882 not found: ID does not exist"
Jul 29 10:59:55.521968 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.521917    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"]
Jul 29 10:59:55.525887 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:55.525857    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0934b-74995788f-j8psd"]
Jul 29 10:59:56.310942 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:56.310898    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" path="/var/lib/kubelet/pods/bc81178b-a296-4689-900c-7411ef6a3af8/volumes"
Jul 29 10:59:57.391676 ip-10-0-129-138 kubenswrapper[2572]: I0729 10:59:57.391636    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:00:02.392588 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:02.392506    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:00:03.400636 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:03.400581    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 29 11:00:03.441020 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:03.440975    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 29 11:00:05.544445 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:05.544360    2572 generic.go:356] "Generic (PLEG): container finished" podID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerID="b578fead8244de118c63f0c9894cc7b8fa71999df2cd4f806eb79dc25e6403a5" exitCode=0
Jul 29 11:00:05.544445 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:05.544405    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" event={"ID":"c16f1604-9687-4840-9dd2-965b42fc9e7f","Type":"ContainerDied","Data":"b578fead8244de118c63f0c9894cc7b8fa71999df2cd4f806eb79dc25e6403a5"}
Jul 29 11:00:05.828995 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:05.828967    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 11:00:05.941343 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:05.941300    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c16f1604-9687-4840-9dd2-965b42fc9e7f-proxy-tls\") pod \"c16f1604-9687-4840-9dd2-965b42fc9e7f\" (UID: \"c16f1604-9687-4840-9dd2-965b42fc9e7f\") "
Jul 29 11:00:05.941534 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:05.941368    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c16f1604-9687-4840-9dd2-965b42fc9e7f-openshift-service-ca-bundle\") pod \"c16f1604-9687-4840-9dd2-965b42fc9e7f\" (UID: \"c16f1604-9687-4840-9dd2-965b42fc9e7f\") "
Jul 29 11:00:05.941726 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:05.941696    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16f1604-9687-4840-9dd2-965b42fc9e7f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c16f1604-9687-4840-9dd2-965b42fc9e7f" (UID: "c16f1604-9687-4840-9dd2-965b42fc9e7f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:00:05.943476 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:05.943447    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16f1604-9687-4840-9dd2-965b42fc9e7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c16f1604-9687-4840-9dd2-965b42fc9e7f" (UID: "c16f1604-9687-4840-9dd2-965b42fc9e7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:00:06.042071 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:06.042032    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c16f1604-9687-4840-9dd2-965b42fc9e7f-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:00:06.042071 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:06.042063    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c16f1604-9687-4840-9dd2-965b42fc9e7f-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:00:06.548934 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:06.548891    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"
Jul 29 11:00:06.548934 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:06.548915    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh" event={"ID":"c16f1604-9687-4840-9dd2-965b42fc9e7f","Type":"ContainerDied","Data":"aa52567b9bdeb6ccee1f39e7fefc0fc00675e39e94a5192c169eaebf5920ecd6"}
Jul 29 11:00:06.549407 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:06.548988    2572 scope.go:117] "RemoveContainer" containerID="b578fead8244de118c63f0c9894cc7b8fa71999df2cd4f806eb79dc25e6403a5"
Jul 29 11:00:06.562570 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:06.562539    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"]
Jul 29 11:00:06.566380 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:06.566352    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f5066-5dd8b587f8-6xmdh"]
Jul 29 11:00:08.309627 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:08.309591    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" path="/var/lib/kubelet/pods/c16f1604-9687-4840-9dd2-965b42fc9e7f/volumes"
Jul 29 11:00:13.400070 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:13.400037    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 11:00:13.440872 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:13.440829    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 29 11:00:23.441083 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:23.441051    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 11:00:25.005135 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005096    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"]
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005378    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005389    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005399    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005404    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005412    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005418    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005425    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kube-rbac-proxy"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005430    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kube-rbac-proxy"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005475    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc81178b-a296-4689-900c-7411ef6a3af8" containerName="sequence-graph-0934b"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005482    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c16f1604-9687-4840-9dd2-965b42fc9e7f" containerName="splitter-graph-f5066"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005487    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kube-rbac-proxy"
Jul 29 11:00:25.005552 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.005493    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="159f0b39-a911-48cc-b0a2-8034742e990a" containerName="kserve-container"
Jul 29 11:00:25.008218 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.008202    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:25.010244 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.010218    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-965a2-kube-rbac-proxy-sar-config\""
Jul 29 11:00:25.010367 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.010218    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-965a2-serving-cert\""
Jul 29 11:00:25.018186 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.018166    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"]
Jul 29 11:00:25.086488 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.086435    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-proxy-tls\") pod \"switch-graph-965a2-5ddc4d9f8b-lpck6\" (UID: \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\") " pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:25.086705 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.086566    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-openshift-service-ca-bundle\") pod \"switch-graph-965a2-5ddc4d9f8b-lpck6\" (UID: \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\") " pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:25.187903 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.187865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-proxy-tls\") pod \"switch-graph-965a2-5ddc4d9f8b-lpck6\" (UID: \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\") " pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:25.188088 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.187962    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-openshift-service-ca-bundle\") pod \"switch-graph-965a2-5ddc4d9f8b-lpck6\" (UID: \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\") " pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:25.188088 ip-10-0-129-138 kubenswrapper[2572]: E0729 11:00:25.188050    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-965a2-serving-cert: secret "switch-graph-965a2-serving-cert" not found
Jul 29 11:00:25.188197 ip-10-0-129-138 kubenswrapper[2572]: E0729 11:00:25.188131    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-proxy-tls podName:2fab34aa-33e0-436f-b2ba-c4bd93f9ee90 nodeName:}" failed. No retries permitted until 2026-07-29 11:00:25.688114893 +0000 UTC m=+1907.916361421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-proxy-tls") pod "switch-graph-965a2-5ddc4d9f8b-lpck6" (UID: "2fab34aa-33e0-436f-b2ba-c4bd93f9ee90") : secret "switch-graph-965a2-serving-cert" not found
Jul 29 11:00:25.188628 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.188604    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-openshift-service-ca-bundle\") pod \"switch-graph-965a2-5ddc4d9f8b-lpck6\" (UID: \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\") " pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:25.692674 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.692631    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-proxy-tls\") pod \"switch-graph-965a2-5ddc4d9f8b-lpck6\" (UID: \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\") " pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:25.695262 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.695231    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-proxy-tls\") pod \"switch-graph-965a2-5ddc4d9f8b-lpck6\" (UID: \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\") " pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:25.919342 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:25.919276    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:26.059806 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:26.059770    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"]
Jul 29 11:00:26.064592 ip-10-0-129-138 kubenswrapper[2572]: W0729 11:00:26.064560    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fab34aa_33e0_436f_b2ba_c4bd93f9ee90.slice/crio-2b57dea63c2c754709d1d56865c5a02efd96f4107bc215ebf68c430d2eca97e0 WatchSource:0}: Error finding container 2b57dea63c2c754709d1d56865c5a02efd96f4107bc215ebf68c430d2eca97e0: Status 404 returned error can't find the container with id 2b57dea63c2c754709d1d56865c5a02efd96f4107bc215ebf68c430d2eca97e0
Jul 29 11:00:26.622651 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:26.622615    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" event={"ID":"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90","Type":"ContainerStarted","Data":"21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f"}
Jul 29 11:00:26.622854 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:26.622670    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" event={"ID":"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90","Type":"ContainerStarted","Data":"2b57dea63c2c754709d1d56865c5a02efd96f4107bc215ebf68c430d2eca97e0"}
Jul 29 11:00:26.622854 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:26.622696    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:26.638357 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:26.638304    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" podStartSLOduration=2.638287227 podStartE2EDuration="2.638287227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:00:26.636675196 +0000 UTC m=+1908.864921745" watchObservedRunningTime="2026-07-29 11:00:26.638287227 +0000 UTC m=+1908.866533818"
Jul 29 11:00:32.631854 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:32.631820    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:00:35.431548 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.431504    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"]
Jul 29 11:00:35.434878 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.434857    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:00:35.436669 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.436645    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-afa2d-kube-rbac-proxy-sar-config\""
Jul 29 11:00:35.436669 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.436658    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-afa2d-serving-cert\""
Jul 29 11:00:35.442797 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.442766    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"]
Jul 29 11:00:35.576457 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.576416    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057666b6-b812-479b-8970-3f6c4a5d3dd3-proxy-tls\") pod \"splitter-graph-afa2d-86d55575d-mkw79\" (UID: \"057666b6-b812-479b-8970-3f6c4a5d3dd3\") " pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:00:35.576618 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.576472    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/057666b6-b812-479b-8970-3f6c4a5d3dd3-openshift-service-ca-bundle\") pod \"splitter-graph-afa2d-86d55575d-mkw79\" (UID: \"057666b6-b812-479b-8970-3f6c4a5d3dd3\") " pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:00:35.677039 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.677002    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057666b6-b812-479b-8970-3f6c4a5d3dd3-proxy-tls\") pod \"splitter-graph-afa2d-86d55575d-mkw79\" (UID: \"057666b6-b812-479b-8970-3f6c4a5d3dd3\") " pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:00:35.677221 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.677047    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/057666b6-b812-479b-8970-3f6c4a5d3dd3-openshift-service-ca-bundle\") pod \"splitter-graph-afa2d-86d55575d-mkw79\" (UID: \"057666b6-b812-479b-8970-3f6c4a5d3dd3\") " pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:00:35.677661 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.677631    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/057666b6-b812-479b-8970-3f6c4a5d3dd3-openshift-service-ca-bundle\") pod \"splitter-graph-afa2d-86d55575d-mkw79\" (UID: \"057666b6-b812-479b-8970-3f6c4a5d3dd3\") " pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:00:35.679456 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.679435    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057666b6-b812-479b-8970-3f6c4a5d3dd3-proxy-tls\") pod \"splitter-graph-afa2d-86d55575d-mkw79\" (UID: \"057666b6-b812-479b-8970-3f6c4a5d3dd3\") " pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:00:35.747490 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.747395    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:00:35.880949 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:35.880902    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"]
Jul 29 11:00:35.884034 ip-10-0-129-138 kubenswrapper[2572]: W0729 11:00:35.884001    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057666b6_b812_479b_8970_3f6c4a5d3dd3.slice/crio-491eb5235d84ec2f0b8b1e11c15829f93ea71cdb84f645543bc00f36d2424590 WatchSource:0}: Error finding container 491eb5235d84ec2f0b8b1e11c15829f93ea71cdb84f645543bc00f36d2424590: Status 404 returned error can't find the container with id 491eb5235d84ec2f0b8b1e11c15829f93ea71cdb84f645543bc00f36d2424590
Jul 29 11:00:36.662215 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:36.662182    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" event={"ID":"057666b6-b812-479b-8970-3f6c4a5d3dd3","Type":"ContainerStarted","Data":"600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086"}
Jul 29 11:00:36.662628 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:36.662233    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" event={"ID":"057666b6-b812-479b-8970-3f6c4a5d3dd3","Type":"ContainerStarted","Data":"491eb5235d84ec2f0b8b1e11c15829f93ea71cdb84f645543bc00f36d2424590"}
Jul 29 11:00:36.662628 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:36.662313    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:00:36.678049 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:36.677989    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" podStartSLOduration=1.677966804 podStartE2EDuration="1.677966804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:00:36.676726109 +0000 UTC m=+1918.904972659" watchObservedRunningTime="2026-07-29 11:00:36.677966804 +0000 UTC m=+1918.906213356"
Jul 29 11:00:42.671650 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:00:42.671619    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:08:49.933632 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:49.933582    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"]
Jul 29 11:08:49.936089 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:49.933875    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d" containerID="cri-o://600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086" gracePeriod=30
Jul 29 11:08:50.067592 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:50.067543    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"]
Jul 29 11:08:50.067856 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:50.067801    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container" containerID="cri-o://a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21" gracePeriod=30
Jul 29 11:08:50.068042 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:50.067820    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kube-rbac-proxy" containerID="cri-o://15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2" gracePeriod=30
Jul 29 11:08:50.485010 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:50.484972    2572 generic.go:356] "Generic (PLEG): container finished" podID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerID="15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2" exitCode=2
Jul 29 11:08:50.485206 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:50.485038    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" event={"ID":"91dea421-e029-486a-a81b-cc4e6f1b0a60","Type":"ContainerDied","Data":"15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2"}
Jul 29 11:08:52.669641 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:52.669595    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:08:53.311554 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.311527    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 11:08:53.384391 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.384354    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-rbac-proxy-sar-config\") pod \"91dea421-e029-486a-a81b-cc4e6f1b0a60\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") "
Jul 29 11:08:53.384557 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.384454    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nrbf\" (UniqueName: \"kubernetes.io/projected/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-api-access-6nrbf\") pod \"91dea421-e029-486a-a81b-cc4e6f1b0a60\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") "
Jul 29 11:08:53.384557 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.384489    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dea421-e029-486a-a81b-cc4e6f1b0a60-proxy-tls\") pod \"91dea421-e029-486a-a81b-cc4e6f1b0a60\" (UID: \"91dea421-e029-486a-a81b-cc4e6f1b0a60\") "
Jul 29 11:08:53.384765 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.384740    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91dea421-e029-486a-a81b-cc4e6f1b0a60" (UID: "91dea421-e029-486a-a81b-cc4e6f1b0a60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:08:53.386685 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.386644    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91dea421-e029-486a-a81b-cc4e6f1b0a60-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91dea421-e029-486a-a81b-cc4e6f1b0a60" (UID: "91dea421-e029-486a-a81b-cc4e6f1b0a60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:08:53.386685 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.386659    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-api-access-6nrbf" (OuterVolumeSpecName: "kube-api-access-6nrbf") pod "91dea421-e029-486a-a81b-cc4e6f1b0a60" (UID: "91dea421-e029-486a-a81b-cc4e6f1b0a60"). InnerVolumeSpecName "kube-api-access-6nrbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:08:53.485709 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.485649    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrbf\" (UniqueName: \"kubernetes.io/projected/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-api-access-6nrbf\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:08:53.485709 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.485700    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91dea421-e029-486a-a81b-cc4e6f1b0a60-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:08:53.485709 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.485712    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91dea421-e029-486a-a81b-cc4e6f1b0a60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:08:53.497124 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.497088    2572 generic.go:356] "Generic (PLEG): container finished" podID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerID="a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21" exitCode=0
Jul 29 11:08:53.497254 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.497147    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" event={"ID":"91dea421-e029-486a-a81b-cc4e6f1b0a60","Type":"ContainerDied","Data":"a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21"}
Jul 29 11:08:53.497254 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.497152    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"
Jul 29 11:08:53.497254 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.497229    2572 scope.go:117] "RemoveContainer" containerID="15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2"
Jul 29 11:08:53.497365 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.497219    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2" event={"ID":"91dea421-e029-486a-a81b-cc4e6f1b0a60","Type":"ContainerDied","Data":"866318d660a0d75f657a7d2617a5e3c768c776912087da7555b8ff77b159de66"}
Jul 29 11:08:53.505734 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.505715    2572 scope.go:117] "RemoveContainer" containerID="a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21"
Jul 29 11:08:53.513514 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.513497    2572 scope.go:117] "RemoveContainer" containerID="15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2"
Jul 29 11:08:53.513784 ip-10-0-129-138 kubenswrapper[2572]: E0729 11:08:53.513762    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2\": container with ID starting with 15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2 not found: ID does not exist" containerID="15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2"
Jul 29 11:08:53.513834 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.513793    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2"} err="failed to get container status \"15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2\": rpc error: code = NotFound desc = could not find container \"15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2\": container with ID starting with 15a43c306bc9afd61b62000098855d09fafe55f3b5d1788fc468b41ccd741ec2 not found: ID does not exist"
Jul 29 11:08:53.513834 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.513811    2572 scope.go:117] "RemoveContainer" containerID="a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21"
Jul 29 11:08:53.514048 ip-10-0-129-138 kubenswrapper[2572]: E0729 11:08:53.514032    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21\": container with ID starting with a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21 not found: ID does not exist" containerID="a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21"
Jul 29 11:08:53.514105 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.514052    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21"} err="failed to get container status \"a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21\": rpc error: code = NotFound desc = could not find container \"a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21\": container with ID starting with a426aab25757b170509afbd3e90e1cec80ee23e4a20ddcf8613d7ed68bdb2b21 not found: ID does not exist"
Jul 29 11:08:53.517335 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.517313    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"]
Jul 29 11:08:53.521522 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:53.521501    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-afa2d-predictor-54c8dbf4d6-k86d2"]
Jul 29 11:08:54.309298 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:54.309258    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" path="/var/lib/kubelet/pods/91dea421-e029-486a-a81b-cc4e6f1b0a60/volumes"
Jul 29 11:08:57.669959 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:08:57.669901    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:09:02.669302 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:02.669253    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:09:02.669704 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:02.669384    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:09:07.669818 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:07.669771    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:09:12.669363 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:12.669316    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:09:17.669340 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:17.669292    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:09:20.079012 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.078985    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:09:20.200127 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.200033    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057666b6-b812-479b-8970-3f6c4a5d3dd3-proxy-tls\") pod \"057666b6-b812-479b-8970-3f6c4a5d3dd3\" (UID: \"057666b6-b812-479b-8970-3f6c4a5d3dd3\") "
Jul 29 11:09:20.200127 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.200120    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/057666b6-b812-479b-8970-3f6c4a5d3dd3-openshift-service-ca-bundle\") pod \"057666b6-b812-479b-8970-3f6c4a5d3dd3\" (UID: \"057666b6-b812-479b-8970-3f6c4a5d3dd3\") "
Jul 29 11:09:20.200508 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.200478    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057666b6-b812-479b-8970-3f6c4a5d3dd3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "057666b6-b812-479b-8970-3f6c4a5d3dd3" (UID: "057666b6-b812-479b-8970-3f6c4a5d3dd3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:09:20.202289 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.202265    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057666b6-b812-479b-8970-3f6c4a5d3dd3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "057666b6-b812-479b-8970-3f6c4a5d3dd3" (UID: "057666b6-b812-479b-8970-3f6c4a5d3dd3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:09:20.300865 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.300809    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/057666b6-b812-479b-8970-3f6c4a5d3dd3-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:09:20.300865 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.300858    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/057666b6-b812-479b-8970-3f6c4a5d3dd3-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:09:20.597375 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.597288    2572 generic.go:356] "Generic (PLEG): container finished" podID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerID="600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086" exitCode=0
Jul 29 11:09:20.597375 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.597337    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"
Jul 29 11:09:20.597375 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.597351    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" event={"ID":"057666b6-b812-479b-8970-3f6c4a5d3dd3","Type":"ContainerDied","Data":"600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086"}
Jul 29 11:09:20.597642 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.597396    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79" event={"ID":"057666b6-b812-479b-8970-3f6c4a5d3dd3","Type":"ContainerDied","Data":"491eb5235d84ec2f0b8b1e11c15829f93ea71cdb84f645543bc00f36d2424590"}
Jul 29 11:09:20.597642 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.597418    2572 scope.go:117] "RemoveContainer" containerID="600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086"
Jul 29 11:09:20.605651 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.605633    2572 scope.go:117] "RemoveContainer" containerID="600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086"
Jul 29 11:09:20.605960 ip-10-0-129-138 kubenswrapper[2572]: E0729 11:09:20.605937    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086\": container with ID starting with 600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086 not found: ID does not exist" containerID="600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086"
Jul 29 11:09:20.606038 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.605974    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086"} err="failed to get container status \"600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086\": rpc error: code = NotFound desc = could not find container \"600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086\": container with ID starting with 600ae664f48eb44557f5dc4218a2ee62a156520d6675d719065b57e0e883c086 not found: ID does not exist"
Jul 29 11:09:20.612205 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.612181    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"]
Jul 29 11:09:20.615276 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:20.615255    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-afa2d-86d55575d-mkw79"]
Jul 29 11:09:22.309814 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:09:22.309779    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" path="/var/lib/kubelet/pods/057666b6-b812-479b-8970-3f6c4a5d3dd3/volumes"
Jul 29 11:16:44.373020 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:44.372914    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"]
Jul 29 11:16:44.375596 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:44.373212    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2" containerID="cri-o://21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f" gracePeriod=30
Jul 29 11:16:44.538868 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:44.538821    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"]
Jul 29 11:16:44.539178 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:44.539150    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container" containerID="cri-o://f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007" gracePeriod=30
Jul 29 11:16:44.539270 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:44.539233    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kube-rbac-proxy" containerID="cri-o://09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7" gracePeriod=30
Jul 29 11:16:45.193806 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:45.193770    2572 generic.go:356] "Generic (PLEG): container finished" podID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerID="09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7" exitCode=2
Jul 29 11:16:45.194028 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:45.193845    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" event={"ID":"43d2a27d-baba-45b7-a758-cbe3a4ee264c","Type":"ContainerDied","Data":"09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7"}
Jul 29 11:16:47.594559 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.594529    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 11:16:47.630792 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.630756    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:16:47.736739 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.736644    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-rbac-proxy-sar-config\") pod \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") "
Jul 29 11:16:47.736739 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.736696    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d2a27d-baba-45b7-a758-cbe3a4ee264c-proxy-tls\") pod \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") "
Jul 29 11:16:47.736739 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.736716    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt6cv\" (UniqueName: \"kubernetes.io/projected/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-api-access-jt6cv\") pod \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\" (UID: \"43d2a27d-baba-45b7-a758-cbe3a4ee264c\") "
Jul 29 11:16:47.737131 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.737099    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43d2a27d-baba-45b7-a758-cbe3a4ee264c" (UID: "43d2a27d-baba-45b7-a758-cbe3a4ee264c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:16:47.738836 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.738803    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d2a27d-baba-45b7-a758-cbe3a4ee264c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43d2a27d-baba-45b7-a758-cbe3a4ee264c" (UID: "43d2a27d-baba-45b7-a758-cbe3a4ee264c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:16:47.738970 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.738944    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-api-access-jt6cv" (OuterVolumeSpecName: "kube-api-access-jt6cv") pod "43d2a27d-baba-45b7-a758-cbe3a4ee264c" (UID: "43d2a27d-baba-45b7-a758-cbe3a4ee264c"). InnerVolumeSpecName "kube-api-access-jt6cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:16:47.837504 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.837466    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:16:47.837504 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.837494    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d2a27d-baba-45b7-a758-cbe3a4ee264c-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:16:47.837504 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:47.837504    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt6cv\" (UniqueName: \"kubernetes.io/projected/43d2a27d-baba-45b7-a758-cbe3a4ee264c-kube-api-access-jt6cv\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:16:48.205660 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.205622    2572 generic.go:356] "Generic (PLEG): container finished" podID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerID="f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007" exitCode=0
Jul 29 11:16:48.205874 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.205677    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"
Jul 29 11:16:48.205874 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.205681    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" event={"ID":"43d2a27d-baba-45b7-a758-cbe3a4ee264c","Type":"ContainerDied","Data":"f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007"}
Jul 29 11:16:48.205874 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.205744    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz" event={"ID":"43d2a27d-baba-45b7-a758-cbe3a4ee264c","Type":"ContainerDied","Data":"c01c2dbcd7788f4c7eeef86835be69208f01fed75752757b9ec0e95bf2394ecc"}
Jul 29 11:16:48.205874 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.205777    2572 scope.go:117] "RemoveContainer" containerID="09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7"
Jul 29 11:16:48.215107 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.215077    2572 scope.go:117] "RemoveContainer" containerID="f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007"
Jul 29 11:16:48.223638 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.223620    2572 scope.go:117] "RemoveContainer" containerID="09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7"
Jul 29 11:16:48.223929 ip-10-0-129-138 kubenswrapper[2572]: E0729 11:16:48.223898    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7\": container with ID starting with 09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7 not found: ID does not exist" containerID="09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7"
Jul 29 11:16:48.223979 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.223944    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7"} err="failed to get container status \"09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7\": rpc error: code = NotFound desc = could not find container \"09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7\": container with ID starting with 09276ed654b4b82202b822be0fccb4b7ac829d38ebfb2b59e0085603f613fba7 not found: ID does not exist"
Jul 29 11:16:48.223979 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.223963    2572 scope.go:117] "RemoveContainer" containerID="f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007"
Jul 29 11:16:48.224222 ip-10-0-129-138 kubenswrapper[2572]: E0729 11:16:48.224203    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007\": container with ID starting with f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007 not found: ID does not exist" containerID="f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007"
Jul 29 11:16:48.224270 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.224227    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007"} err="failed to get container status \"f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007\": rpc error: code = NotFound desc = could not find container \"f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007\": container with ID starting with f2f8769b808766fee1ce12ad1713e13f16e65817a660e408660f912f437d7007 not found: ID does not exist"
Jul 29 11:16:48.228834 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.228805    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"]
Jul 29 11:16:48.234606 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.234580    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-965a2-predictor-64fd5c48d4-rjkhz"]
Jul 29 11:16:48.309723 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:48.309689    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" path="/var/lib/kubelet/pods/43d2a27d-baba-45b7-a758-cbe3a4ee264c/volumes"
Jul 29 11:16:52.630333 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:52.630291    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:16:57.629871 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:57.629826    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:16:57.630349 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:16:57.629986    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:17:02.364555 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:02.364525    2572 ???:1] "http: TLS handshake error from 10.0.137.188:58048: EOF"
Jul 29 11:17:02.366967 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:02.366936    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:02.629714 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:02.629669    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:17:03.196799 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:03.196674    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:03.995037 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:03.995001    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:04.792604 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:04.792569    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:05.582518 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:05.582484    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:06.372951 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:06.372906    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:07.145621 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:07.145593    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:07.630432 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:07.630390    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:17:07.966897 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:07.966807    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:08.799423 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:08.799394    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:09.622095 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:09.622066    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:10.406642 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:10.406613    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:11.202001 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:11.201973    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:12.005164 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:12.005136    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:12.630280 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:12.630232    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 11:17:12.829948 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:12.829904    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-965a2-5ddc4d9f8b-lpck6_2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/switch-graph-965a2/0.log"
Jul 29 11:17:15.017852 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.017824    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:17:15.133217 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.133179    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-proxy-tls\") pod \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\" (UID: \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\") "
Jul 29 11:17:15.133427 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.133239    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-openshift-service-ca-bundle\") pod \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\" (UID: \"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90\") "
Jul 29 11:17:15.133653 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.133620    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" (UID: "2fab34aa-33e0-436f-b2ba-c4bd93f9ee90"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:17:15.135366 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.135340    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" (UID: "2fab34aa-33e0-436f-b2ba-c4bd93f9ee90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:17:15.234610 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.234508    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-openshift-service-ca-bundle\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:17:15.234610 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.234547    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90-proxy-tls\") on node \"ip-10-0-129-138.ec2.internal\" DevicePath \"\""
Jul 29 11:17:15.306545 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.306509    2572 generic.go:356] "Generic (PLEG): container finished" podID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerID="21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f" exitCode=0
Jul 29 11:17:15.306723 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.306555    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"
Jul 29 11:17:15.306723 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.306592    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" event={"ID":"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90","Type":"ContainerDied","Data":"21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f"}
Jul 29 11:17:15.306723 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.306640    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6" event={"ID":"2fab34aa-33e0-436f-b2ba-c4bd93f9ee90","Type":"ContainerDied","Data":"2b57dea63c2c754709d1d56865c5a02efd96f4107bc215ebf68c430d2eca97e0"}
Jul 29 11:17:15.306723 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.306662    2572 scope.go:117] "RemoveContainer" containerID="21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f"
Jul 29 11:17:15.314635 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.314615    2572 scope.go:117] "RemoveContainer" containerID="21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f"
Jul 29 11:17:15.315067 ip-10-0-129-138 kubenswrapper[2572]: E0729 11:17:15.315042    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f\": container with ID starting with 21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f not found: ID does not exist" containerID="21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f"
Jul 29 11:17:15.315151 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.315076    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f"} err="failed to get container status \"21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f\": rpc error: code = NotFound desc = could not find container \"21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f\": container with ID starting with 21a1ff1fa0659a6502fca111555474be5869347d3b3144fc9639620ed793f32f not found: ID does not exist"
Jul 29 11:17:15.325907 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.325877    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"]
Jul 29 11:17:15.329164 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:15.329135    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-965a2-5ddc4d9f8b-lpck6"]
Jul 29 11:17:16.308902 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:16.308870    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" path="/var/lib/kubelet/pods/2fab34aa-33e0-436f-b2ba-c4bd93f9ee90/volumes"
Jul 29 11:17:17.710913 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.710875    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8kqzn/must-gather-xtxbg"]
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711162    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kube-rbac-proxy"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711173    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kube-rbac-proxy"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711181    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711187    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711193    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711200    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711210    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711216    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711225    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kube-rbac-proxy"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711231    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kube-rbac-proxy"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711240    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711245    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711284    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kserve-container"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711291    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kserve-container"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711296    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="057666b6-b812-479b-8970-3f6c4a5d3dd3" containerName="splitter-graph-afa2d"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711303    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d2a27d-baba-45b7-a758-cbe3a4ee264c" containerName="kube-rbac-proxy"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711308    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="91dea421-e029-486a-a81b-cc4e6f1b0a60" containerName="kube-rbac-proxy"
Jul 29 11:17:17.711332 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.711315    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fab34aa-33e0-436f-b2ba-c4bd93f9ee90" containerName="switch-graph-965a2"
Jul 29 11:17:17.715499 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.715478    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kqzn/must-gather-xtxbg"
Jul 29 11:17:17.717734 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.717714    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8kqzn\"/\"kube-root-ca.crt\""
Jul 29 11:17:17.718061 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.718040    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8kqzn\"/\"default-dockercfg-9wrc4\""
Jul 29 11:17:17.718316 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.718298    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8kqzn\"/\"openshift-service-ca.crt\""
Jul 29 11:17:17.722984 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.722959    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kqzn/must-gather-xtxbg"]
Jul 29 11:17:17.755662 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.755623    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2162956e-659b-4183-9181-7d9da2d6da22-must-gather-output\") pod \"must-gather-xtxbg\" (UID: \"2162956e-659b-4183-9181-7d9da2d6da22\") " pod="openshift-must-gather-8kqzn/must-gather-xtxbg"
Jul 29 11:17:17.755856 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.755682    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb97s\" (UniqueName: \"kubernetes.io/projected/2162956e-659b-4183-9181-7d9da2d6da22-kube-api-access-cb97s\") pod \"must-gather-xtxbg\" (UID: \"2162956e-659b-4183-9181-7d9da2d6da22\") " pod="openshift-must-gather-8kqzn/must-gather-xtxbg"
Jul 29 11:17:17.856968 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.856901    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2162956e-659b-4183-9181-7d9da2d6da22-must-gather-output\") pod \"must-gather-xtxbg\" (UID: \"2162956e-659b-4183-9181-7d9da2d6da22\") " pod="openshift-must-gather-8kqzn/must-gather-xtxbg"
Jul 29 11:17:17.857118 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.856989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb97s\" (UniqueName: \"kubernetes.io/projected/2162956e-659b-4183-9181-7d9da2d6da22-kube-api-access-cb97s\") pod \"must-gather-xtxbg\" (UID: \"2162956e-659b-4183-9181-7d9da2d6da22\") " pod="openshift-must-gather-8kqzn/must-gather-xtxbg"
Jul 29 11:17:17.857278 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.857257    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2162956e-659b-4183-9181-7d9da2d6da22-must-gather-output\") pod \"must-gather-xtxbg\" (UID: \"2162956e-659b-4183-9181-7d9da2d6da22\") " pod="openshift-must-gather-8kqzn/must-gather-xtxbg"
Jul 29 11:17:17.865147 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:17.865117    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb97s\" (UniqueName: \"kubernetes.io/projected/2162956e-659b-4183-9181-7d9da2d6da22-kube-api-access-cb97s\") pod \"must-gather-xtxbg\" (UID: \"2162956e-659b-4183-9181-7d9da2d6da22\") " pod="openshift-must-gather-8kqzn/must-gather-xtxbg"
Jul 29 11:17:18.025060 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:18.024967    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kqzn/must-gather-xtxbg"
Jul 29 11:17:18.159119 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:18.159084    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kqzn/must-gather-xtxbg"]
Jul 29 11:17:18.162027 ip-10-0-129-138 kubenswrapper[2572]: W0729 11:17:18.161996    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2162956e_659b_4183_9181_7d9da2d6da22.slice/crio-5a5f6b409245301e192a952b0c280da1e5570879a2e6042274742dd65279c694 WatchSource:0}: Error finding container 5a5f6b409245301e192a952b0c280da1e5570879a2e6042274742dd65279c694: Status 404 returned error can't find the container with id 5a5f6b409245301e192a952b0c280da1e5570879a2e6042274742dd65279c694
Jul 29 11:17:18.163997 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:18.163976    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 11:17:18.318794 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:18.318708    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kqzn/must-gather-xtxbg" event={"ID":"2162956e-659b-4183-9181-7d9da2d6da22","Type":"ContainerStarted","Data":"5a5f6b409245301e192a952b0c280da1e5570879a2e6042274742dd65279c694"}
Jul 29 11:17:21.334221 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:21.334175    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kqzn/must-gather-xtxbg" event={"ID":"2162956e-659b-4183-9181-7d9da2d6da22","Type":"ContainerStarted","Data":"d76c94de2d8b07edb6842e43487d0113923ac667adf5d0a53d447663bea33522"}
Jul 29 11:17:21.334739 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:21.334243    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kqzn/must-gather-xtxbg" event={"ID":"2162956e-659b-4183-9181-7d9da2d6da22","Type":"ContainerStarted","Data":"a95c6149f956bd2dc28524395c2dcb6367fbb0779dc90c14246697bafc15d1e5"}
Jul 29 11:17:21.350247 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:21.350194    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8kqzn/must-gather-xtxbg" podStartSLOduration=2.022635651 podStartE2EDuration="4.350176869s" totalImagesPullingTime="2.327541218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:17:21.349711049 +0000 UTC m=+2923.577957602" watchObservedRunningTime="2026-07-29 11:17:21.350176869 +0000 UTC m=+2923.578423419"
Jul 29 11:17:21.996946 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:21.996888    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fcpjf_b2f0ed09-a52b-4305-8fa6-435794cc0ff3/global-pull-secret-syncer/0.log"
Jul 29 11:17:22.178558 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:22.178506    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jrqzr_d900008f-ab70-4ed3-8521-a2098f40ae4c/konnectivity-agent/0.log"
Jul 29 11:17:22.252188 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:22.252100    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-138.ec2.internal_a3e4af55a69b4ca5f48ea6f06887aafd/haproxy/0.log"
Jul 29 11:17:26.779987 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:26.779872    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gwdgm_f113b545-977d-4f0e-93b0-8bdd345dbfe5/node-exporter/0.log"
Jul 29 11:17:26.806877 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:26.806816    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gwdgm_f113b545-977d-4f0e-93b0-8bdd345dbfe5/kube-rbac-proxy/0.log"
Jul 29 11:17:26.843530 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:26.843357    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gwdgm_f113b545-977d-4f0e-93b0-8bdd345dbfe5/init-textfile/0.log"
Jul 29 11:17:30.553752 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.553713    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"]
Jul 29 11:17:30.559124 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.559094    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.566169 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.566096    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"]
Jul 29 11:17:30.673887 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.673843    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-lib-modules\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.674114 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.673954    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-podres\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.674114 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.674007    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-956tl\" (UniqueName: \"kubernetes.io/projected/79660cba-fd4f-4064-a490-57887819cd32-kube-api-access-956tl\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.674114 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.674037    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-proc\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.674114 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.674071    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-sys\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.775101 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.775062    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-podres\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.775294 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.775117    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-956tl\" (UniqueName: \"kubernetes.io/projected/79660cba-fd4f-4064-a490-57887819cd32-kube-api-access-956tl\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.775294 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.775140    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-proc\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.775294 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.775163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-sys\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.775294 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.775232    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-sys\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.775294 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.775240    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-podres\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.775294 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.775240    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-proc\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.775294 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.775278    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-lib-modules\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.775630 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.775377    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79660cba-fd4f-4064-a490-57887819cd32-lib-modules\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.783199 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.783170    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-956tl\" (UniqueName: \"kubernetes.io/projected/79660cba-fd4f-4064-a490-57887819cd32-kube-api-access-956tl\") pod \"perf-node-gather-daemonset-wcnvr\" (UID: \"79660cba-fd4f-4064-a490-57887819cd32\") " pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:30.874076 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:30.874042    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:31.041526 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:31.041493    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"]
Jul 29 11:17:31.045562 ip-10-0-129-138 kubenswrapper[2572]: W0729 11:17:31.045535    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod79660cba_fd4f_4064_a490_57887819cd32.slice/crio-527c3328e17ce750d93574eeaf6c92f8883399faa54b037e2570a50ecb65ab18 WatchSource:0}: Error finding container 527c3328e17ce750d93574eeaf6c92f8883399faa54b037e2570a50ecb65ab18: Status 404 returned error can't find the container with id 527c3328e17ce750d93574eeaf6c92f8883399faa54b037e2570a50ecb65ab18
Jul 29 11:17:31.385503 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:31.385415    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr" event={"ID":"79660cba-fd4f-4064-a490-57887819cd32","Type":"ContainerStarted","Data":"4571f971ad02d19ae880883938557ff49f422b305664ccedf6285472d5d49255"}
Jul 29 11:17:31.385503 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:31.385475    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr" event={"ID":"79660cba-fd4f-4064-a490-57887819cd32","Type":"ContainerStarted","Data":"527c3328e17ce750d93574eeaf6c92f8883399faa54b037e2570a50ecb65ab18"}
Jul 29 11:17:31.385748 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:31.385521    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:31.403743 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:31.403687    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr" podStartSLOduration=1.403663334 podStartE2EDuration="1.403663334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 11:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:17:31.403011123 +0000 UTC m=+2933.631257676" watchObservedRunningTime="2026-07-29 11:17:31.403663334 +0000 UTC m=+2933.631909884"
Jul 29 11:17:31.765677 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:31.765569    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmftl_1124f153-b0c0-4132-88a3-c60068f9e2c3/dns/0.log"
Jul 29 11:17:31.791843 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:31.791795    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmftl_1124f153-b0c0-4132-88a3-c60068f9e2c3/kube-rbac-proxy/0.log"
Jul 29 11:17:31.853160 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:31.853132    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fns2s_8799dbf0-e093-403b-997d-6bc3b2323a96/dns-node-resolver/0.log"
Jul 29 11:17:32.372247 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:32.372210    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5df59cb6-7ncb4_6d461338-d5de-41de-ba00-45dd33b27b02/registry/0.log"
Jul 29 11:17:32.451681 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:32.451647    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xgkvk_a2f82d39-8f16-4606-a6d5-8f9ced950f85/node-ca/0.log"
Jul 29 11:17:33.622178 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:33.622146    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g7l9f_c8c3349f-e8de-4229-93c5-8aa5499e3a72/serve-healthcheck-canary/0.log"
Jul 29 11:17:34.229061 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:34.229024    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rwhkx_090b067b-9649-4758-934b-459ca2bb383d/kube-rbac-proxy/0.log"
Jul 29 11:17:34.253621 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:34.253588    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rwhkx_090b067b-9649-4758-934b-459ca2bb383d/exporter/0.log"
Jul 29 11:17:34.277969 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:34.277940    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rwhkx_090b067b-9649-4758-934b-459ca2bb383d/extractor/0.log"
Jul 29 11:17:36.662535 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:36.662505    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-w9cz2_1868c5d5-3cfe-4553-8b21-78a5eadb082b/server/0.log"
Jul 29 11:17:36.773665 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:36.773628    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-6xvjw_ae28219a-e857-41fb-96cc-fd277c228c04/manager/0.log"
Jul 29 11:17:36.872109 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:36.872071    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-cmzvx_d9980c2e-b5cc-4d41-9481-52814816f0ed/seaweedfs/0.log"
Jul 29 11:17:37.399629 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:37.399594    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8kqzn/perf-node-gather-daemonset-wcnvr"
Jul 29 11:17:42.897268 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:42.897235    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c4t8m_56c584cc-8c25-427e-9209-d70836610d73/kube-multus-additional-cni-plugins/0.log"
Jul 29 11:17:42.922701 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:42.922666    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c4t8m_56c584cc-8c25-427e-9209-d70836610d73/egress-router-binary-copy/0.log"
Jul 29 11:17:42.946757 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:42.946728    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c4t8m_56c584cc-8c25-427e-9209-d70836610d73/cni-plugins/0.log"
Jul 29 11:17:42.975080 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:42.975048    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c4t8m_56c584cc-8c25-427e-9209-d70836610d73/bond-cni-plugin/0.log"
Jul 29 11:17:43.002280 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:43.002248    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c4t8m_56c584cc-8c25-427e-9209-d70836610d73/routeoverride-cni/0.log"
Jul 29 11:17:43.027438 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:43.027411    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c4t8m_56c584cc-8c25-427e-9209-d70836610d73/whereabouts-cni-bincopy/0.log"
Jul 29 11:17:43.052972 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:43.052935    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c4t8m_56c584cc-8c25-427e-9209-d70836610d73/whereabouts-cni/0.log"
Jul 29 11:17:43.275202 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:43.275119    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p44gh_779f2766-886b-4d01-9fdc-b54073e8604b/kube-multus/0.log"
Jul 29 11:17:43.446872 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:43.446843    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l8kkx_7be3cb65-5879-412b-b438-99e55616cf3e/network-metrics-daemon/0.log"
Jul 29 11:17:43.471574 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:43.471547    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l8kkx_7be3cb65-5879-412b-b438-99e55616cf3e/kube-rbac-proxy/0.log"
Jul 29 11:17:44.293909 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:44.293878    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2hvgf_b3b896e1-b046-4eea-a864-6169f95a415f/ovn-controller/0.log"
Jul 29 11:17:44.315404 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:44.315366    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2hvgf_b3b896e1-b046-4eea-a864-6169f95a415f/ovn-acl-logging/0.log"
Jul 29 11:17:44.341168 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:44.341126    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2hvgf_b3b896e1-b046-4eea-a864-6169f95a415f/kube-rbac-proxy-node/0.log"
Jul 29 11:17:44.372019 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:44.371993    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2hvgf_b3b896e1-b046-4eea-a864-6169f95a415f/kube-rbac-proxy-ovn-metrics/0.log"
Jul 29 11:17:44.397724 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:44.397695    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2hvgf_b3b896e1-b046-4eea-a864-6169f95a415f/northd/0.log"
Jul 29 11:17:44.423726 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:44.423700    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2hvgf_b3b896e1-b046-4eea-a864-6169f95a415f/nbdb/0.log"
Jul 29 11:17:44.448308 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:44.448268    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2hvgf_b3b896e1-b046-4eea-a864-6169f95a415f/sbdb/0.log"
Jul 29 11:17:44.502895 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:44.502865    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2hvgf_b3b896e1-b046-4eea-a864-6169f95a415f/ovnkube-controller/0.log"
Jul 29 11:17:46.113313 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:46.113278    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wjnft_084f30e4-3825-48a8-96ca-c5ebc0c10b5a/network-check-target-container/0.log"
Jul 29 11:17:47.126163 ip-10-0-129-138 kubenswrapper[2572]: I0729 11:17:47.126092    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-l5szg_e8471251-f5f4-48f8-91d6-7864fbc9db94/iptables-alerter/0.log"