Jul 07 14:09:08.892958 ip-10-0-138-82 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.224460    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228400    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228413    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228418    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228421    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228425    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228428    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228431    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228434    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:09.385896 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228437    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:09.299560 ip-10-0-138-82 systemd[1]: Started Kubernetes Kubelet.
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228440    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228443    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228445    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228448    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228451    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228454    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228456    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228459    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228461    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228468    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228471    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228474    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228476    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228478    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228482    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228485    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228488    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228490    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:09.387687 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228493    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228496    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228499    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228502    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228505    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228507    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228509    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228512    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228514    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228517    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228520    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228522    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228525    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228527    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228530    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228532    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228537    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228540    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228544    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:09.389356 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228547    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228550    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228553    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228555    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228558    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228560    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228563    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228566    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228569    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228571    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228574    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228577    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228579    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228582    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228584    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228587    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228589    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228592    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228594    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228597    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:09.390388 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228599    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228603    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228606    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228609    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228628    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228633    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228636    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228639    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228642    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228645    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228648    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228651    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228657    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228660    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228663    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228666    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228669    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228671    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228674    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:09.391482 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228677    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228680    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228682    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228685    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228688    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228690    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228693    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228695    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228698    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228701    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228703    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228706    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228708    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.228711    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229406    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229414    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229419    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229422    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229425    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229429    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:09.392547 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229432    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229435    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229438    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229441    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229444    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229446    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229449    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229452    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229454    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229457    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229460    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229462    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229464    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229467    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229469    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229472    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229474    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229477    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229479    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229482    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:09.393631 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229484    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229487    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229490    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229492    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229495    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229498    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229500    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229515    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229518    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229536    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229540    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229543    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229546    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229549    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229552    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229555    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229558    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229563    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:09.395121 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229567    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229571    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229573    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229576    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229579    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229581    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229584    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229587    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229589    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229592    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229594    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229597    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229599    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229602    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229605    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229608    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229612    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229626    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229629    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229635    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:09.396009 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229638    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229641    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229643    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229646    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229648    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229651    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229654    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229656    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229659    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229662    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229664    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229667    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229669    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229672    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229675    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229678    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229680    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229683    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229685    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229688    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:09.397097 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229690    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229693    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229695    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229698    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229700    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229703    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229705    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229708    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229710    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229713    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229716    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229719    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229722    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229724    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.229727    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230177    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230182    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230185    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230189    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230193    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:09.398197 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230196    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230199    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230202    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230205    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230208    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230211    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230214    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230217    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230219    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230221    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230224    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230226    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230229    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230231    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230235    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230238    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230240    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230242    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230245    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230247    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:09.399522 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230250    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230253    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230255    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230259    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230261    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230264    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230267    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230270    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230273    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230276    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230279    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230282    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230284    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230287    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230290    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230292    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230295    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230298    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230300    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230304    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:09.400405 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230306    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230308    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230311    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230314    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230316    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230319    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230323    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230325    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230328    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230331    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230333    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230335    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230338    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230341    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230343    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230347    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230350    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230352    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230354    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:09.401495 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230357    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230359    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230362    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230364    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230367    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230369    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230372    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230375    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230377    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230380    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230382    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230385    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230387    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230390    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230392    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230395    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230397    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230401    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230404    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230408    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:09.402396 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230411    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230413    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230416    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230418    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230420    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230423    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230425    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230428    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230432    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230435    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230437    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230440    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230442    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230445    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.230447    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230511    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230518    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230524    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230528    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230532    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230537    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 14:09:09.403502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230544    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230551    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230555    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230560    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230565    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230569    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230573    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230576    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230579    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230582    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230586    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230589    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230593    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230596    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230599    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230602    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230606    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230610    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230613    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230630    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230641    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230644    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230647    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230650    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 14:09:09.404391 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230654    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230657    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230662    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230665    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230668    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230671    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230674    2565 flags.go:64] FLAG: --enable-server="true"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230677    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230686    2565 flags.go:64] FLAG: --event-burst="100"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230689    2565 flags.go:64] FLAG: --event-qps="50"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230692    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230695    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230698    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230702    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230705    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230708    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230711    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230714    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230718    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230721    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230724    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230727    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230729    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230732    2565 flags.go:64] FLAG: --feature-gates=""
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230736    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 14:09:09.406130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230739    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230742    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230746    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230749    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230752    2565 flags.go:64] FLAG: --help="false"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230755    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230758    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230761    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230764    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230767    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230770    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230773    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230776    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230778    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230781    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230784    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230787    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230790    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230793    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230796    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230799    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230801    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230805    2565 flags.go:64] FLAG: --lock-file=""
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230807    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 14:09:09.407223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230811    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230816    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230822    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230825    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230828    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230831    2565 flags.go:64] FLAG: --logging-format="text"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230834    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230837    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230839    2565 flags.go:64] FLAG: --manifest-url=""
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230842    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230848    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230851    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230855    2565 flags.go:64] FLAG: --max-pods="110"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230858    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230861    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230864    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230867    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230870    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230873    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230876    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230887    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230890    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230893    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 14:09:09.411217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230896    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230899    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230904    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230907    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230910    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230913    2565 flags.go:64] FLAG: --port="10250"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230916    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230919    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02a6539242e6706ee"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230923    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230926    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230929    2565 flags.go:64] FLAG: --register-node="true"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230933    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230936    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230939    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230942    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230945    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230949    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230952    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230954    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230958    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230962    2565 flags.go:64] FLAG: --runonce="false"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230965    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230968    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230971    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230974    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 14:09:09.434062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230977    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230980    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230983    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230987    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230990    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230993    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230996    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.230998    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231001    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231005    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231007    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231013    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231016    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231018    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231022    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231025    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231028    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231031    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231035    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231038    2565 flags.go:64] FLAG: --v="2"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231042    2565 flags.go:64] FLAG: --version="false"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231046    2565 flags.go:64] FLAG: --vmodule=""
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231050    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.231053    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231174    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:09.434873 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231178    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231183    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231185    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231188    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231190    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231193    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231197    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231200    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231203    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231205    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231208    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231210    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231213    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231216    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231218    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231221    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231223    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231226    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231228    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:09.435650 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231231    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231233    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231236    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231239    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231241    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231256    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231261    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231264    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231266    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231269    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231272    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231274    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231277    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231280    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231287    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231290    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231292    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231295    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231298    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231300    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:09.436205 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231303    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231305    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231308    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231311    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231314    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231316    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231319    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231322    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231324    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231327    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231330    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231332    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231335    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231337    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231340    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231342    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231345    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231347    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231352    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231355    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:09.436824 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231357    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231360    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231362    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231365    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231368    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231370    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231374    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231376    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231379    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231381    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231384    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231386    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231389    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231391    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231394    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231397    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231399    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231402    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231406    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:09.437526 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231409    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231413    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231416    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231418    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231421    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231424    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231427    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231430    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231434    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231437    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231440    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231445    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231448    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231451    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231454    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231457    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231459    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231462    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231464    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:09.438131 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.231468    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.232032    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.241343    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.241361    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.241386    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.241392    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241470    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241475    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241479    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241482    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241485    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241488    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241491    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241494    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241497    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:09.438739 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241500    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241502    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241505    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241507    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241510    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241514    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241518    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241521    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241524    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241529    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241532    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241535    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241538    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241542    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241544    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241547    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241550    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241552    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241555    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:09.439468 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241558    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241561    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241564    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241566    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241569    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241571    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241574    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241577    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241580    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241582    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241585    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241588    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241590    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241593    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241596    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241598    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241601    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241604    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241606    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241609    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:09.723096 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241612    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241634    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241640    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241643    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241646    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241649    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241652    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241654    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241659    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241661    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241664    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241667    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241670    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241672    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241675    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241677    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241680    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241682    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241685    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241687    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:09.724146 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241690    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241693    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241695    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241698    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241700    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241703    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241705    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241708    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241710    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241713    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241716    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241719    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241721    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241724    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241728    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241731    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241733    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241736    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241738    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:09.724886 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241741    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241745    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241748    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241750    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241753    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241755    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241758    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241761    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241763    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241766    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241768    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241771    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.241776    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241911    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241916    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 14:09:09.725464 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241920    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241922    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241925    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241928    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241931    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241934    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241937    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241940    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241943    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241946    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241949    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241952    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241954    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241959    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241961    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241964    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241967    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241969    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241974    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 14:09:09.726016 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241976    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241979    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241983    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241987    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241990    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241993    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241995    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.241998    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242001    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242003    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242006    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242008    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242011    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242014    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242018    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242022    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242024    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242027    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242030    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 14:09:09.726656 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242033    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242036    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242039    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242041    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242044    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242047    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242050    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242054    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242057    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242060    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242062    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242065    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242069    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242071    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242074    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242077    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242079    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242082    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242084    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242087    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242089    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 14:09:09.727284 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242092    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242094    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242097    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242099    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242102    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242104    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242106    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242109    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242112    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242114    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242116    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242119    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242121    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242124    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242126    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242129    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242132    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242135    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242139    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242141    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 14:09:09.727903 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242144    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242146    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242149    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242153    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242155    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242158    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242160    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242163    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242166    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242169    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242171    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242174    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242176    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242179    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242183    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242186    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242189    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 14:09:09.728634 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:09.242191    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.242196    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.242314    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.244349    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.245568    2565 server.go:1019] "Starting client certificate rotation"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.245674    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.245747    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.247908    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.249166    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.259708    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.263223    2565 log.go:25] "Validated CRI v1 image API"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.265326    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 14:09:09.729148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.267427    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a8128278-f8a2-4d5c-817c-0b57be650360:/dev/nvme0n1p4 c0e6d93d-76a6-485a-b9b9-5b2c0b619be2:/dev/nvme0n1p3]
Jul 07 14:09:09.729691 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.267447    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 14:09:09.729691 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.275468    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.278009    2565 manager.go:216] Machine: {Timestamp:2026-07-07 14:09:09.276846942 +0000 UTC m=+0.274437649 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3066640 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec264af472890ae4fadce389cc49c22f SystemUUID:ec264af4-7289-0ae4-fadc-e389cc49c22f BootID:5e7a1e46-5cd5-4c6f-b4f4-e1c6696eb025 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b9:00:0d:ca:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b9:00:0d:ca:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:1c:ef:e1:68:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.278118    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.278211    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.281028    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.281057    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.281685    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.281696    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.281704    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.281720    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.282995    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.286762    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.286952    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.286983    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.286995    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.287009    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.287020    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.287030    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 14:09:09.730137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.287493    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.287509    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.289572    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.291429    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.291456    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292505    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292518    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292525    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292531    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292540    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292545    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292554    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292560    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292567    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292573    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292581    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.292586    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.293194    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.293200    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.296872    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.297135    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.297447    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.298642    2565 server.go:1268] "Started kubelet"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.298739    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.298789    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.298874    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.300535    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.301476    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.304104    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.305038    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-82.ec2.internal.18c0072ea1871eb5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-82.ec2.internal,UID:ip-10-0-138-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-82.ec2.internal,},FirstTimestamp:2026-07-07 14:09:09.298593461 +0000 UTC m=+0.296184167,LastTimestamp:2026-07-07 14:09:09.298593461 +0000 UTC m=+0.296184167,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-82.ec2.internal,}"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.307289    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lcpv5"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.307986    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.307993    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.308213    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.308580    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.308596    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.308714    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.308735    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.308711    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.308935    2565 factory.go:55] Registering systemd factory
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.308952    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 07 14:09:09.730734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.309152    2565 factory.go:153] Registering CRI-O factory
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.309163    2565 factory.go:223] Registration of the crio container factory successfully
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.309222    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.309244    2565 factory.go:103] Registering Raw factory
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.309254    2565 manager.go:1196] Started watching for new ooms in manager
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.309866    2565 manager.go:349] Starting recovery of all containers
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.315949    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lcpv5"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.318429    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.318508    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.321139    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.323256    2565 manager.go:354] Recovery completed
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.324906    2565 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.329693    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.331430    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.331457    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.331467    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.332019    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.332031    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.332034    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.332137    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.332696    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.332721    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.332732    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.334243    2565 policy_none.go:49] "None policy: Start"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.334270    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.334281    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.336424    2565 policy_none.go:47] "Start"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.373449    2565 manager.go:337] "Starting Device Plugin manager"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.373483    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.373495    2565 server.go:87] "Starting device plugin registration server"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.373768    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.373783    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.373860    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.373935    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.373944    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.374396    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.374441    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.474192    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.475243    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.475276    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.475286    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.475312    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.480261    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.480300    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.480320    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 14:09:09.951904 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.480355    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.482997    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.483665    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.483687    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-82.ec2.internal\": node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.515745    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.580840    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal"]
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.580936    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.582571    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.582595    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.582605    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.585007    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.585117    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.585148    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.585849    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.585877    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.585892    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.585894    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.585913    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.585922    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.588141    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.588164    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.588870    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.588897    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.588907    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeHasSufficientPID"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.615984    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.616044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50cab86694ee5c9634e52b1e02eef957-config\") pod \"kube-apiserver-proxy-ip-10-0-138-82.ec2.internal\" (UID: \"50cab86694ee5c9634e52b1e02eef957\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.616067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30b9a95d424cb79735acebdc49ec05f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"30b9a95d424cb79735acebdc49ec05f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.616085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30b9a95d424cb79735acebdc49ec05f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"30b9a95d424cb79735acebdc49ec05f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.620486    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-82.ec2.internal\" not found" node="ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.625109    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-82.ec2.internal\" not found" node="ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.716353    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:09.954038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.716413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50cab86694ee5c9634e52b1e02eef957-config\") pod \"kube-apiserver-proxy-ip-10-0-138-82.ec2.internal\" (UID: \"50cab86694ee5c9634e52b1e02eef957\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.956433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.716438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30b9a95d424cb79735acebdc49ec05f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"30b9a95d424cb79735acebdc49ec05f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.956433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.716454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30b9a95d424cb79735acebdc49ec05f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"30b9a95d424cb79735acebdc49ec05f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.956433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.716508    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50cab86694ee5c9634e52b1e02eef957-config\") pod \"kube-apiserver-proxy-ip-10-0-138-82.ec2.internal\" (UID: \"50cab86694ee5c9634e52b1e02eef957\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.956433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.716512    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30b9a95d424cb79735acebdc49ec05f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"30b9a95d424cb79735acebdc49ec05f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.956433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.716539    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30b9a95d424cb79735acebdc49ec05f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal\" (UID: \"30b9a95d424cb79735acebdc49ec05f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.956433 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.816954    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:09.956433 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:09.917297    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:09.956433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.923448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:09.956433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:09.928341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:10.189505 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:10.018332    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:10.189505 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:10.118778    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:10.418758 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:10.219103    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:10.418758 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.245240    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 14:09:10.418758 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.245374    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 14:09:10.418758 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.319073    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:04:09 +0000 UTC" deadline="2028-03-27 11:56:45.022293786 +0000 UTC"
Jul 07 14:09:10.418758 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.319114    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15093h47m34.703183251s"
Jul 07 14:09:10.418758 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:10.319161    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:10.419989 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:10.419641    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-82.ec2.internal\" not found"
Jul 07 14:09:10.651043 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.448477    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:10.651043 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.509107    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:10.651043 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.519899    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:09:10.651043 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.521369    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal"
Jul 07 14:09:10.651043 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.530516    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 14:09:10.651043 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.572187    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:10.652273 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.652100    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:10.653438 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:10.653410    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cab86694ee5c9634e52b1e02eef957.slice/crio-526db31cb834c3df3a36a1808a05b75600ea8c9d99f0ab3719ec201208b7099a WatchSource:0}: Error finding container 526db31cb834c3df3a36a1808a05b75600ea8c9d99f0ab3719ec201208b7099a: Status 404 returned error can't find the container with id 526db31cb834c3df3a36a1808a05b75600ea8c9d99f0ab3719ec201208b7099a
Jul 07 14:09:10.665901 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:10.665882    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:09:10.672081 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:10.672055    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b9a95d424cb79735acebdc49ec05f0.slice/crio-bb7b0cbc0e3511c747001c5490c63e54329390a2d31e526f4184e51cd2a7571a WatchSource:0}: Error finding container bb7b0cbc0e3511c747001c5490c63e54329390a2d31e526f4184e51cd2a7571a: Status 404 returned error can't find the container with id bb7b0cbc0e3511c747001c5490c63e54329390a2d31e526f4184e51cd2a7571a
Jul 07 14:09:11.288497 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.288456    2565 apiserver.go:52] "Watching apiserver"
Jul 07 14:09:11.298711 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.298674    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 14:09:11.300443 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.300259    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-dh95c","kube-system/konnectivity-agent-p66vc","kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal","openshift-multus/multus-additional-cni-plugins-54wn6","openshift-multus/network-metrics-daemon-znqh7","openshift-network-operator/iptables-alerter-bhjlt","openshift-ovn-kubernetes/ovnkube-node-82xpw","kube-system/global-pull-secret-syncer-m6ghz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45","openshift-cluster-node-tuning-operator/tuned-kdp5h","openshift-dns/node-resolver-gj9qr","openshift-image-registry/node-ca-9xdbg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal","openshift-multus/multus-xtl8q"]
Jul 07 14:09:11.303198 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.303171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.305519 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.305493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.305993 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.305969    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 14:09:11.306764 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.306737    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 14:09:11.306868 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.306851    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 14:09:11.306932 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.306866    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 14:09:11.307050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.307003    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 14:09:11.307050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.307003    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6t9jr\""
Jul 07 14:09:11.307234 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.307205    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 14:09:11.307424 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.307402    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 14:09:11.307851 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.307829    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.308099 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.308075    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-25dtn\""
Jul 07 14:09:11.308368 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.308354    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 14:09:11.308727 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.308697    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 14:09:11.310007 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.309909    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 14:09:11.310007 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.309991    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ft4c9\""
Jul 07 14:09:11.310156 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.310057    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 14:09:11.310156 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.310152    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 14:09:11.310325 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.310305    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 14:09:11.310537 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.310517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.312659 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.312640    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j27zg\""
Jul 07 14:09:11.312768 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.312649    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 14:09:11.312768 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.312696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:11.312907 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.312768    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 14:09:11.312907 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.312762    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:11.314993 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.314974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.317324 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.317305    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.317767 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.317535    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xpt77\""
Jul 07 14:09:11.317767 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.317634    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 14:09:11.317767 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.317747    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 14:09:11.317959 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.317778    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:09:11.318121 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.318093    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 14:09:11.319115 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.319099    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 14:09:11.319972 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.319950    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 14:09:11.320071 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.320030    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:11.320133 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.320101    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:11.321216 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.321196    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5srvs\""
Jul 07 14:09:11.321294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.321215    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 14:09:11.323190 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-run-ovn\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.323282 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-hostroot\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.323282 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-run-systemd\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.323282 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-cni-netd\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.323282 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.323472 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b291786-af74-43e3-84e4-2af9581a134f-ovn-node-metrics-cert\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.323472 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323386    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfgkg\" (UniqueName: \"kubernetes.io/projected/7b291786-af74-43e3-84e4-2af9581a134f-kube-api-access-rfgkg\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.323472 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323412    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:11.323647 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchmr\" (UniqueName: \"kubernetes.io/projected/d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed-kube-api-access-lchmr\") pod \"node-resolver-gj9qr\" (UID: \"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed\") " pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.323711 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c78112d6-bdd7-44c5-9214-04e37f5c16a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.323764 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c78112d6-bdd7-44c5-9214-04e37f5c16a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.323764 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-etc-openvswitch\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.323863 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-cnibin\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.323863 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-os-release\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.323863 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-run-netns\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.323863 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323859    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.324050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-run-netns\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323914    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-run-openvswitch\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b291786-af74-43e3-84e4-2af9581a134f-ovnkube-script-lib\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.323976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46a39168-6bed-493d-a1e0-c4313508adc4-cni-binary-copy\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324004    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-var-lib-cni-multus\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324042    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46a39168-6bed-493d-a1e0-c4313508adc4-multus-daemon-config\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-kubelet\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-log-socket\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-cni-bin\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324137    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-system-cni-dir\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-slash\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-run-ovn-kubernetes\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-run-k8s-cni-cncf-io\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324230    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-var-lib-cni-bin\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-run-multus-certs\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpb9m\" (UniqueName: \"kubernetes.io/projected/46a39168-6bed-493d-a1e0-c4313508adc4-kube-api-access-zpb9m\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6rj\" (UniqueName: \"kubernetes.io/projected/c78112d6-bdd7-44c5-9214-04e37f5c16a2-kube-api-access-5g6rj\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-var-lib-openvswitch\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-node-log\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b291786-af74-43e3-84e4-2af9581a134f-env-overrides\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed-hosts-file\") pod \"node-resolver-gj9qr\" (UID: \"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed\") " pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324498    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-system-cni-dir\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-multus-cni-dir\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-multus-conf-dir\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-etc-kubernetes\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-systemd-units\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b291786-af74-43e3-84e4-2af9581a134f-ovnkube-config\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.324747 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed-tmp-dir\") pod \"node-resolver-gj9qr\" (UID: \"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed\") " pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.325207 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-multus-socket-dir-parent\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.325207 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-var-lib-kubelet\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.325207 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-cnibin\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.325207 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-os-release\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.325207 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.324942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c78112d6-bdd7-44c5-9214-04e37f5c16a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.325704 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.325684    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 14:09:11.325704 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.325701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.325863 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.325704    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jsxvv\""
Jul 07 14:09:11.326037 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.326017    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 14:09:11.327902 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.327879    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 14:09:11.327985 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.327959    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 14:09:11.328270 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.328245    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:11.328376 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.328314    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:11.328440 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.328414    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zh9wx\""
Jul 07 14:09:11.331135 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.331118    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.333916 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.333693    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 14:09:11.333916 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.333696    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 14:09:11.333916 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.333734    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 14:09:11.333916 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.333746    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l5r75\""
Jul 07 14:09:11.343400 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.343376    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zlx5f"
Jul 07 14:09:11.353654 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.353627    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zlx5f"
Jul 07 14:09:11.410010 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.409979    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 14:09:11.425323 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-run-multus-certs\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-systemd\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-var-lib-openvswitch\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425406    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-var-lib-openvswitch\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425414    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-run-multus-certs\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425473    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-node-log\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-node-log\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b291786-af74-43e3-84e4-2af9581a134f-env-overrides\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-system-cni-dir\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-multus-conf-dir\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425564    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-etc-kubernetes\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b291786-af74-43e3-84e4-2af9581a134f-ovnkube-config\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed-tmp-dir\") pod \"node-resolver-gj9qr\" (UID: \"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed\") " pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425644    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-multus-conf-dir\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425660    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-system-cni-dir\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-var-lib-kubelet\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-cnibin\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425715    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-etc-kubernetes\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.426239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c78112d6-bdd7-44c5-9214-04e37f5c16a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425764    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-var-lib-kubelet\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ee39c5e-114a-4756-b5e3-f68d0d72c49b-iptables-alerter-script\") pod \"iptables-alerter-bhjlt\" (UID: \"4ee39c5e-114a-4756-b5e3-f68d0d72c49b\") " pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425797    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-cnibin\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppvrj\" (UniqueName: \"kubernetes.io/projected/0ae7a068-2e12-4564-b34f-e8eee03f70bd-kube-api-access-ppvrj\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-sys\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-lib-modules\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-run-systemd\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-cni-netd\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b291786-af74-43e3-84e4-2af9581a134f-ovn-node-metrics-cert\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425932    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-run-systemd\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c78112d6-bdd7-44c5-9214-04e37f5c16a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd47c6c1-367e-416b-8c25-b64c869d5c22-serviceca\") pod \"node-ca-9xdbg\" (UID: \"fd47c6c1-367e-416b-8c25-b64c869d5c22\") " pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425973    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-cni-netd\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.425994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-host\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426001    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed-tmp-dir\") pod \"node-resolver-gj9qr\" (UID: \"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed\") " pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-etc-openvswitch\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.427225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-cnibin\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426077    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-os-release\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-sys-fs\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426105    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b291786-af74-43e3-84e4-2af9581a134f-env-overrides\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46a39168-6bed-493d-a1e0-c4313508adc4-cni-binary-copy\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-var-lib-cni-multus\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-kubelet\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426192    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-os-release\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-cni-bin\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426201    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-cnibin\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426121    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-etc-openvswitch\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-system-cni-dir\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426241    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-cni-bin\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426267    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-dbus\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426312    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-var-lib-cni-multus\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/177c9d3c-7f21-498a-b90d-1c818b61a216-tmp\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.428040 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426305    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-system-cni-dir\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426264    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b291786-af74-43e3-84e4-2af9581a134f-ovnkube-config\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426358    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c78112d6-bdd7-44c5-9214-04e37f5c16a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426371    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-kubelet\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpb9m\" (UniqueName: \"kubernetes.io/projected/46a39168-6bed-493d-a1e0-c4313508adc4-kube-api-access-zpb9m\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426388    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6rj\" (UniqueName: \"kubernetes.io/projected/c78112d6-bdd7-44c5-9214-04e37f5c16a2-kube-api-access-5g6rj\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-sysconfig\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-os-release\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65chb\" (UniqueName: \"kubernetes.io/projected/4ee39c5e-114a-4756-b5e3-f68d0d72c49b-kube-api-access-65chb\") pod \"iptables-alerter-bhjlt\" (UID: \"4ee39c5e-114a-4756-b5e3-f68d0d72c49b\") " pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426708    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46a39168-6bed-493d-a1e0-c4313508adc4-cni-binary-copy\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-socket-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426759    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-os-release\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed-hosts-file\") pod \"node-resolver-gj9qr\" (UID: \"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed\") " pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-multus-cni-dir\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426921    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed-hosts-file\") pod \"node-resolver-gj9qr\" (UID: \"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed\") " pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.428838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-systemd-units\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426975    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-systemd-units\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.426993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-multus-socket-dir-parent\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427023    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-multus-socket-dir-parent\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-kubelet-config\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427044    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-multus-cni-dir\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-run\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427072    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c78112d6-bdd7-44c5-9214-04e37f5c16a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427107    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-run-ovn\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427133    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-run-ovn\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-hostroot\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427246    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-hostroot\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnnz\" (UniqueName: \"kubernetes.io/projected/a92bcc45-e37c-46ed-9887-6a895f7e7104-kube-api-access-nxnnz\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-modprobe-d\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfgkg\" (UniqueName: \"kubernetes.io/projected/7b291786-af74-43e3-84e4-2af9581a134f-kube-api-access-rfgkg\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.429502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchmr\" (UniqueName: \"kubernetes.io/projected/d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed-kube-api-access-lchmr\") pod \"node-resolver-gj9qr\" (UID: \"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed\") " pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c78112d6-bdd7-44c5-9214-04e37f5c16a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-registration-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-device-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427588    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0b57686-10e6-4348-b196-2a49ace4083f-konnectivity-ca\") pod \"konnectivity-agent-p66vc\" (UID: \"b0b57686-10e6-4348-b196-2a49ace4083f\") " pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-run-netns\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427713    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-sysctl-d\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427866    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c78112d6-bdd7-44c5-9214-04e37f5c16a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-tuned\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxd69\" (UniqueName: \"kubernetes.io/projected/177c9d3c-7f21-498a-b90d-1c818b61a216-kube-api-access-jxd69\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.427993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-run-netns\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-run-openvswitch\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428037    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c78112d6-bdd7-44c5-9214-04e37f5c16a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.430130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b291786-af74-43e3-84e4-2af9581a134f-ovnkube-script-lib\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428038    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-run-netns\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428070    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-run-openvswitch\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46a39168-6bed-493d-a1e0-c4313508adc4-multus-daemon-config\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd47c6c1-367e-416b-8c25-b64c869d5c22-host\") pod \"node-ca-9xdbg\" (UID: \"fd47c6c1-367e-416b-8c25-b64c869d5c22\") " pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjwm\" (UniqueName: \"kubernetes.io/projected/fd47c6c1-367e-416b-8c25-b64c869d5c22-kube-api-access-zjjwm\") pod \"node-ca-9xdbg\" (UID: \"fd47c6c1-367e-416b-8c25-b64c869d5c22\") " pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428142    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-run-netns\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0b57686-10e6-4348-b196-2a49ace4083f-agent-certs\") pod \"konnectivity-agent-p66vc\" (UID: \"b0b57686-10e6-4348-b196-2a49ace4083f\") " pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428213    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-log-socket\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-kubernetes\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-sysctl-conf\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-var-lib-kubelet\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428284    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-log-socket\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ee39c5e-114a-4756-b5e3-f68d0d72c49b-host-slash\") pod \"iptables-alerter-bhjlt\" (UID: \"4ee39c5e-114a-4756-b5e3-f68d0d72c49b\") " pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-slash\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-run-ovn-kubernetes\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.430601 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-run-k8s-cni-cncf-io\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.431147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428392    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-slash\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.431147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428410    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-var-lib-cni-bin\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.431147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428454    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b291786-af74-43e3-84e4-2af9581a134f-host-run-ovn-kubernetes\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.431147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428454    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-run-k8s-cni-cncf-io\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.431147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428490    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46a39168-6bed-493d-a1e0-c4313508adc4-host-var-lib-cni-bin\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.431147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428496    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b291786-af74-43e3-84e4-2af9581a134f-ovnkube-script-lib\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.431147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.428521    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46a39168-6bed-493d-a1e0-c4313508adc4-multus-daemon-config\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.431147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.429889    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b291786-af74-43e3-84e4-2af9581a134f-ovn-node-metrics-cert\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.435239 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.435218    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpb9m\" (UniqueName: \"kubernetes.io/projected/46a39168-6bed-493d-a1e0-c4313508adc4-kube-api-access-zpb9m\") pod \"multus-xtl8q\" (UID: \"46a39168-6bed-493d-a1e0-c4313508adc4\") " pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.435773 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.435744    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6rj\" (UniqueName: \"kubernetes.io/projected/c78112d6-bdd7-44c5-9214-04e37f5c16a2-kube-api-access-5g6rj\") pod \"multus-additional-cni-plugins-54wn6\" (UID: \"c78112d6-bdd7-44c5-9214-04e37f5c16a2\") " pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.436500 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.436480    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfgkg\" (UniqueName: \"kubernetes.io/projected/7b291786-af74-43e3-84e4-2af9581a134f-kube-api-access-rfgkg\") pod \"ovnkube-node-82xpw\" (UID: \"7b291786-af74-43e3-84e4-2af9581a134f\") " pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.441433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.441411    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchmr\" (UniqueName: \"kubernetes.io/projected/d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed-kube-api-access-lchmr\") pod \"node-resolver-gj9qr\" (UID: \"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed\") " pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.485721 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.485645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" event={"ID":"30b9a95d424cb79735acebdc49ec05f0","Type":"ContainerStarted","Data":"bb7b0cbc0e3511c747001c5490c63e54329390a2d31e526f4184e51cd2a7571a"}
Jul 07 14:09:11.486717 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.486692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" event={"ID":"50cab86694ee5c9634e52b1e02eef957","Type":"ContainerStarted","Data":"526db31cb834c3df3a36a1808a05b75600ea8c9d99f0ab3719ec201208b7099a"}
Jul 07 14:09:11.529359 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.529359 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-registration-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.529359 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-device-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0b57686-10e6-4348-b196-2a49ace4083f-konnectivity-ca\") pod \"konnectivity-agent-p66vc\" (UID: \"b0b57686-10e6-4348-b196-2a49ace4083f\") " pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-sysctl-d\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-tuned\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxd69\" (UniqueName: \"kubernetes.io/projected/177c9d3c-7f21-498a-b90d-1c818b61a216-kube-api-access-jxd69\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529466    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529481    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-device-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529465    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-registration-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd47c6c1-367e-416b-8c25-b64c869d5c22-host\") pod \"node-ca-9xdbg\" (UID: \"fd47c6c1-367e-416b-8c25-b64c869d5c22\") " pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529523    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd47c6c1-367e-416b-8c25-b64c869d5c22-host\") pod \"node-ca-9xdbg\" (UID: \"fd47c6c1-367e-416b-8c25-b64c869d5c22\") " pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjwm\" (UniqueName: \"kubernetes.io/projected/fd47c6c1-367e-416b-8c25-b64c869d5c22-kube-api-access-zjjwm\") pod \"node-ca-9xdbg\" (UID: \"fd47c6c1-367e-416b-8c25-b64c869d5c22\") " pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0b57686-10e6-4348-b196-2a49ace4083f-agent-certs\") pod \"konnectivity-agent-p66vc\" (UID: \"b0b57686-10e6-4348-b196-2a49ace4083f\") " pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-kubernetes\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529638    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-sysctl-d\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-sysctl-conf\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.529682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-var-lib-kubelet\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ee39c5e-114a-4756-b5e3-f68d0d72c49b-host-slash\") pod \"iptables-alerter-bhjlt\" (UID: \"4ee39c5e-114a-4756-b5e3-f68d0d72c49b\") " pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-systemd\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529763    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-sysctl-conf\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ee39c5e-114a-4756-b5e3-f68d0d72c49b-iptables-alerter-script\") pod \"iptables-alerter-bhjlt\" (UID: \"4ee39c5e-114a-4756-b5e3-f68d0d72c49b\") " pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529794    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-var-lib-kubelet\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppvrj\" (UniqueName: \"kubernetes.io/projected/0ae7a068-2e12-4564-b34f-e8eee03f70bd-kube-api-access-ppvrj\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-sys\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-lib-modules\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd47c6c1-367e-416b-8c25-b64c869d5c22-serviceca\") pod \"node-ca-9xdbg\" (UID: \"fd47c6c1-367e-416b-8c25-b64c869d5c22\") " pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.529886    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-host\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-sys-fs\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.529982    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret podName:e8ae7e96-9ae1-45b4-a74a-fb810990dd5c nodeName:}" failed. No retries permitted until 2026-07-07 14:09:12.029942104 +0000 UTC m=+3.027532820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret") pod "global-pull-secret-syncer-m6ghz" (UID: "e8ae7e96-9ae1-45b4-a74a-fb810990dd5c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.529996    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-sys-fs\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-dbus\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:11.530427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530051    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0b57686-10e6-4348-b196-2a49ace4083f-konnectivity-ca\") pod \"konnectivity-agent-p66vc\" (UID: \"b0b57686-10e6-4348-b196-2a49ace4083f\") " pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/177c9d3c-7f21-498a-b90d-1c818b61a216-tmp\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-sysconfig\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65chb\" (UniqueName: \"kubernetes.io/projected/4ee39c5e-114a-4756-b5e3-f68d0d72c49b-kube-api-access-65chb\") pod \"iptables-alerter-bhjlt\" (UID: \"4ee39c5e-114a-4756-b5e3-f68d0d72c49b\") " pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-socket-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-kubelet-config\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-run\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnnz\" (UniqueName: \"kubernetes.io/projected/a92bcc45-e37c-46ed-9887-6a895f7e7104-kube-api-access-nxnnz\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530313    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ee39c5e-114a-4756-b5e3-f68d0d72c49b-iptables-alerter-script\") pod \"iptables-alerter-bhjlt\" (UID: \"4ee39c5e-114a-4756-b5e3-f68d0d72c49b\") " pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530324    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-modprobe-d\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530404    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-kubelet-config\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530421    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-run\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.530514    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530520    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-lib-modules\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530552    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-kubernetes\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.531272 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.530563    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:12.030545219 +0000 UTC m=+3.028135913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530602    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530665    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-modprobe-d\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530669    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-sysconfig\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530765    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0ae7a068-2e12-4564-b34f-e8eee03f70bd-socket-dir\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530822    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-sys\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530861    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-systemd\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530893    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/177c9d3c-7f21-498a-b90d-1c818b61a216-host\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530911    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ee39c5e-114a-4756-b5e3-f68d0d72c49b-host-slash\") pod \"iptables-alerter-bhjlt\" (UID: \"4ee39c5e-114a-4756-b5e3-f68d0d72c49b\") " pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.530911    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-dbus\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:11.532114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.531388    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd47c6c1-367e-416b-8c25-b64c869d5c22-serviceca\") pod \"node-ca-9xdbg\" (UID: \"fd47c6c1-367e-416b-8c25-b64c869d5c22\") " pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.532643 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.532287    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/177c9d3c-7f21-498a-b90d-1c818b61a216-etc-tuned\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.532643 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.532476    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0b57686-10e6-4348-b196-2a49ace4083f-agent-certs\") pod \"konnectivity-agent-p66vc\" (UID: \"b0b57686-10e6-4348-b196-2a49ace4083f\") " pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:11.532897 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.532876    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/177c9d3c-7f21-498a-b90d-1c818b61a216-tmp\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.542333 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.542009    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:11.542333 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.542036    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:11.542333 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.542050    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9m7fb for pod openshift-network-diagnostics/network-check-target-dh95c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:11.542333 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:11.542108    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb podName:e9fd18af-03b0-434f-9ef3-a9b2775d545f nodeName:}" failed. No retries permitted until 2026-07-07 14:09:12.042090762 +0000 UTC m=+3.039681470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9m7fb" (UniqueName: "kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb") pod "network-check-target-dh95c" (UID: "e9fd18af-03b0-434f-9ef3-a9b2775d545f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:11.543760 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.543703    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxd69\" (UniqueName: \"kubernetes.io/projected/177c9d3c-7f21-498a-b90d-1c818b61a216-kube-api-access-jxd69\") pod \"tuned-kdp5h\" (UID: \"177c9d3c-7f21-498a-b90d-1c818b61a216\") " pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.544969 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.544944    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjwm\" (UniqueName: \"kubernetes.io/projected/fd47c6c1-367e-416b-8c25-b64c869d5c22-kube-api-access-zjjwm\") pod \"node-ca-9xdbg\" (UID: \"fd47c6c1-367e-416b-8c25-b64c869d5c22\") " pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.544969 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.544960    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65chb\" (UniqueName: \"kubernetes.io/projected/4ee39c5e-114a-4756-b5e3-f68d0d72c49b-kube-api-access-65chb\") pod \"iptables-alerter-bhjlt\" (UID: \"4ee39c5e-114a-4756-b5e3-f68d0d72c49b\") " pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.545267 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.545250    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppvrj\" (UniqueName: \"kubernetes.io/projected/0ae7a068-2e12-4564-b34f-e8eee03f70bd-kube-api-access-ppvrj\") pod \"aws-ebs-csi-driver-node-jjm45\" (UID: \"0ae7a068-2e12-4564-b34f-e8eee03f70bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.545479 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.545461    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnnz\" (UniqueName: \"kubernetes.io/projected/a92bcc45-e37c-46ed-9887-6a895f7e7104-kube-api-access-nxnnz\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:11.615756 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.615715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:11.623203 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.623165    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gj9qr"
Jul 07 14:09:11.632169 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.632144    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xtl8q"
Jul 07 14:09:11.637958 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.637939    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-54wn6"
Jul 07 14:09:11.647552 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.647523    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bhjlt"
Jul 07 14:09:11.654377 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.654351    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45"
Jul 07 14:09:11.662132 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.662103    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:11.667949 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.667928    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kdp5h"
Jul 07 14:09:11.673629 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.673594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9xdbg"
Jul 07 14:09:11.756280 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:11.756243    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 14:09:11.785828 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:11.785800    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b57686_10e6_4348_b196_2a49ace4083f.slice/crio-1139266bd0d1d482763298087e993d967d8d1cd4741dcfad9e6250f540c2e43d WatchSource:0}: Error finding container 1139266bd0d1d482763298087e993d967d8d1cd4741dcfad9e6250f540c2e43d: Status 404 returned error can't find the container with id 1139266bd0d1d482763298087e993d967d8d1cd4741dcfad9e6250f540c2e43d
Jul 07 14:09:11.786276 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:11.786244    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd47c6c1_367e_416b_8c25_b64c869d5c22.slice/crio-4c9fa9e35e9a8a194a6f00d9db48ab9100875a2f52ce2be2322eb201478d832d WatchSource:0}: Error finding container 4c9fa9e35e9a8a194a6f00d9db48ab9100875a2f52ce2be2322eb201478d832d: Status 404 returned error can't find the container with id 4c9fa9e35e9a8a194a6f00d9db48ab9100875a2f52ce2be2322eb201478d832d
Jul 07 14:09:11.790991 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:11.790965    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78112d6_bdd7_44c5_9214_04e37f5c16a2.slice/crio-23a887c56f40a02db56499f8e19ceba5922536b2233b50ac8b5d3e9f0f0ad912 WatchSource:0}: Error finding container 23a887c56f40a02db56499f8e19ceba5922536b2233b50ac8b5d3e9f0f0ad912: Status 404 returned error can't find the container with id 23a887c56f40a02db56499f8e19ceba5922536b2233b50ac8b5d3e9f0f0ad912
Jul 07 14:09:11.791499 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:11.791473    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177c9d3c_7f21_498a_b90d_1c818b61a216.slice/crio-d2ca53feaf28ce3c09bb5f93f7abb5b16a2ace3c7d65fd182541f45f7e3a3322 WatchSource:0}: Error finding container d2ca53feaf28ce3c09bb5f93f7abb5b16a2ace3c7d65fd182541f45f7e3a3322: Status 404 returned error can't find the container with id d2ca53feaf28ce3c09bb5f93f7abb5b16a2ace3c7d65fd182541f45f7e3a3322
Jul 07 14:09:11.793601 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:11.793493    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae7a068_2e12_4564_b34f_e8eee03f70bd.slice/crio-959d6fd009c56b2271ae57315587f0c471280bd7ae4f9078764ecd86f7ede29e WatchSource:0}: Error finding container 959d6fd009c56b2271ae57315587f0c471280bd7ae4f9078764ecd86f7ede29e: Status 404 returned error can't find the container with id 959d6fd009c56b2271ae57315587f0c471280bd7ae4f9078764ecd86f7ede29e
Jul 07 14:09:11.794315 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:11.794293    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd185cfb0_b3b3_4ef3_acbe_ea86e5ad0fed.slice/crio-5e9aa650f416b7f2df156f884439ed26438cd9106e1c25ae27a3bc07cbc7f13e WatchSource:0}: Error finding container 5e9aa650f416b7f2df156f884439ed26438cd9106e1c25ae27a3bc07cbc7f13e: Status 404 returned error can't find the container with id 5e9aa650f416b7f2df156f884439ed26438cd9106e1c25ae27a3bc07cbc7f13e
Jul 07 14:09:11.795168 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:11.795147    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a39168_6bed_493d_a1e0_c4313508adc4.slice/crio-e39c43b0c81fee8ce37106a0626011b6612ad7fbff67b9fc40fb5fa5a6c86bb2 WatchSource:0}: Error finding container e39c43b0c81fee8ce37106a0626011b6612ad7fbff67b9fc40fb5fa5a6c86bb2: Status 404 returned error can't find the container with id e39c43b0c81fee8ce37106a0626011b6612ad7fbff67b9fc40fb5fa5a6c86bb2
Jul 07 14:09:11.797512 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:11.797486    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee39c5e_114a_4756_b5e3_f68d0d72c49b.slice/crio-233824093a89d74594521226a67cc75d925ffe80c678115e280d61ae0b68a90e WatchSource:0}: Error finding container 233824093a89d74594521226a67cc75d925ffe80c678115e280d61ae0b68a90e: Status 404 returned error can't find the container with id 233824093a89d74594521226a67cc75d925ffe80c678115e280d61ae0b68a90e
Jul 07 14:09:11.797770 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:11.797749    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b291786_af74_43e3_84e4_2af9581a134f.slice/crio-77eb7ae8e7e2e0aa42740e140b09c51dde955a6a497c2178fba1f0d7013ef3a9 WatchSource:0}: Error finding container 77eb7ae8e7e2e0aa42740e140b09c51dde955a6a497c2178fba1f0d7013ef3a9: Status 404 returned error can't find the container with id 77eb7ae8e7e2e0aa42740e140b09c51dde955a6a497c2178fba1f0d7013ef3a9
Jul 07 14:09:12.033546 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.033509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:12.033719 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.033565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:12.033719 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.033690    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:12.033803 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.033770    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:13.033748398 +0000 UTC m=+4.031339097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:12.033803 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.033690    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:12.033902 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.033869    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret podName:e8ae7e96-9ae1-45b4-a74a-fb810990dd5c nodeName:}" failed. No retries permitted until 2026-07-07 14:09:13.03384796 +0000 UTC m=+4.031438654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret") pod "global-pull-secret-syncer-m6ghz" (UID: "e8ae7e96-9ae1-45b4-a74a-fb810990dd5c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:12.134430 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.134223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:12.134568 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.134370    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:12.134568 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.134529    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:12.134568 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.134550    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9m7fb for pod openshift-network-diagnostics/network-check-target-dh95c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:12.134734 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.134638    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb podName:e9fd18af-03b0-434f-9ef3-a9b2775d545f nodeName:}" failed. No retries permitted until 2026-07-07 14:09:13.134603987 +0000 UTC m=+4.132194685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m7fb" (UniqueName: "kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb") pod "network-check-target-dh95c" (UID: "e9fd18af-03b0-434f-9ef3-a9b2775d545f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:12.357165 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.357039    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:04:11 +0000 UTC" deadline="2028-01-13 05:23:48.738426851 +0000 UTC"
Jul 07 14:09:12.357165 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.357083    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13311h14m36.381348848s"
Jul 07 14:09:12.481266 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.481227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:12.481880 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.481378    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:12.481880 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.481433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:12.481880 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:12.481536    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:12.490285 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.490248    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54wn6" event={"ID":"c78112d6-bdd7-44c5-9214-04e37f5c16a2","Type":"ContainerStarted","Data":"23a887c56f40a02db56499f8e19ceba5922536b2233b50ac8b5d3e9f0f0ad912"}
Jul 07 14:09:12.491851 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.491825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9xdbg" event={"ID":"fd47c6c1-367e-416b-8c25-b64c869d5c22","Type":"ContainerStarted","Data":"4c9fa9e35e9a8a194a6f00d9db48ab9100875a2f52ce2be2322eb201478d832d"}
Jul 07 14:09:12.493680 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.493601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p66vc" event={"ID":"b0b57686-10e6-4348-b196-2a49ace4083f","Type":"ContainerStarted","Data":"1139266bd0d1d482763298087e993d967d8d1cd4741dcfad9e6250f540c2e43d"}
Jul 07 14:09:12.494914 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.494887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gj9qr" event={"ID":"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed","Type":"ContainerStarted","Data":"5e9aa650f416b7f2df156f884439ed26438cd9106e1c25ae27a3bc07cbc7f13e"}
Jul 07 14:09:12.496211 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.496188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45" event={"ID":"0ae7a068-2e12-4564-b34f-e8eee03f70bd","Type":"ContainerStarted","Data":"959d6fd009c56b2271ae57315587f0c471280bd7ae4f9078764ecd86f7ede29e"}
Jul 07 14:09:12.497519 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.497498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kdp5h" event={"ID":"177c9d3c-7f21-498a-b90d-1c818b61a216","Type":"ContainerStarted","Data":"d2ca53feaf28ce3c09bb5f93f7abb5b16a2ace3c7d65fd182541f45f7e3a3322"}
Jul 07 14:09:12.499824 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.499797    2565 generic.go:356] "Generic (PLEG): container finished" podID="30b9a95d424cb79735acebdc49ec05f0" containerID="842e09f6bd06cb9cf5d344d0168129fbaa1bc4f7538758a822ae12c1d573b912" exitCode=0
Jul 07 14:09:12.499904 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.499876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" event={"ID":"30b9a95d424cb79735acebdc49ec05f0","Type":"ContainerDied","Data":"842e09f6bd06cb9cf5d344d0168129fbaa1bc4f7538758a822ae12c1d573b912"}
Jul 07 14:09:12.501524 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.501434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" event={"ID":"7b291786-af74-43e3-84e4-2af9581a134f","Type":"ContainerStarted","Data":"77eb7ae8e7e2e0aa42740e140b09c51dde955a6a497c2178fba1f0d7013ef3a9"}
Jul 07 14:09:12.502921 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.502898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bhjlt" event={"ID":"4ee39c5e-114a-4756-b5e3-f68d0d72c49b","Type":"ContainerStarted","Data":"233824093a89d74594521226a67cc75d925ffe80c678115e280d61ae0b68a90e"}
Jul 07 14:09:12.504180 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:12.504156    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xtl8q" event={"ID":"46a39168-6bed-493d-a1e0-c4313508adc4","Type":"ContainerStarted","Data":"e39c43b0c81fee8ce37106a0626011b6612ad7fbff67b9fc40fb5fa5a6c86bb2"}
Jul 07 14:09:13.042254 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:13.042221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:13.042435 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:13.042285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:13.042435 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:13.042388    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:13.042435 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:13.042392    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:13.042578 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:13.042449    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:15.042430953 +0000 UTC m=+6.040021646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:13.042578 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:13.042465    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret podName:e8ae7e96-9ae1-45b4-a74a-fb810990dd5c nodeName:}" failed. No retries permitted until 2026-07-07 14:09:15.042458201 +0000 UTC m=+6.040048893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret") pod "global-pull-secret-syncer-m6ghz" (UID: "e8ae7e96-9ae1-45b4-a74a-fb810990dd5c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:13.143038 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:13.143006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:13.143242 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:13.143149    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:13.143242 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:13.143165    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:13.143242 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:13.143174    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9m7fb for pod openshift-network-diagnostics/network-check-target-dh95c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:13.143405 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:13.143246    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb podName:e9fd18af-03b0-434f-9ef3-a9b2775d545f nodeName:}" failed. No retries permitted until 2026-07-07 14:09:15.143225045 +0000 UTC m=+6.140815745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m7fb" (UniqueName: "kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb") pod "network-check-target-dh95c" (UID: "e9fd18af-03b0-434f-9ef3-a9b2775d545f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:13.358281 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:13.358165    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:04:11 +0000 UTC" deadline="2028-04-01 07:05:20.785613136 +0000 UTC"
Jul 07 14:09:13.358281 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:13.358206    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15208h56m7.427412215s"
Jul 07 14:09:13.481971 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:13.481217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:13.481971 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:13.481381    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:14.481641 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:14.480907    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:14.481641 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:14.481040    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:14.481641 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:14.481429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:14.481641 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:14.481522    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:14.530289 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:14.529574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9xdbg" event={"ID":"fd47c6c1-367e-416b-8c25-b64c869d5c22","Type":"ContainerStarted","Data":"d47926c802bd34e7f766d083aef7a0abfb42c3f9bb034591d4618bb8c3bec30a"}
Jul 07 14:09:14.546105 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:14.545561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" event={"ID":"30b9a95d424cb79735acebdc49ec05f0","Type":"ContainerStarted","Data":"9ce445500094511d71b44251add7caf8e49b31dc6c9a0ac140f092fbfd4edce8"}
Jul 07 14:09:14.551651 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:14.551406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" event={"ID":"50cab86694ee5c9634e52b1e02eef957","Type":"ContainerStarted","Data":"932d50806e48e1f97288f2324b242411a81f6200f827aecdf31a2d9afae510cf"}
Jul 07 14:09:14.572256 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:14.572191    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9xdbg" podStartSLOduration=3.34185805 podStartE2EDuration="5.57216797s" totalImagesPullingTime="2.23030992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:14.549399061 +0000 UTC m=+5.546989776" watchObservedRunningTime="2026-07-07 14:09:14.57216797 +0000 UTC m=+5.569758687"
Jul 07 14:09:14.621860 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:14.621796    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-82.ec2.internal" podStartSLOduration=4.621772915 podStartE2EDuration="4.621772915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:14.572666645 +0000 UTC m=+5.570257361" watchObservedRunningTime="2026-07-07 14:09:14.621772915 +0000 UTC m=+5.619363631"
Jul 07 14:09:15.062750 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:15.062712    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:15.062950 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:15.062780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:15.062950 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:15.062916    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:15.063063 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:15.062983    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret podName:e8ae7e96-9ae1-45b4-a74a-fb810990dd5c nodeName:}" failed. No retries permitted until 2026-07-07 14:09:19.062961233 +0000 UTC m=+10.060551939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret") pod "global-pull-secret-syncer-m6ghz" (UID: "e8ae7e96-9ae1-45b4-a74a-fb810990dd5c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:15.063379 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:15.063361    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:15.063435 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:15.063416    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:19.063400346 +0000 UTC m=+10.060991043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:15.165928 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:15.163650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:15.165928 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:15.163830    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:15.165928 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:15.163860    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:15.165928 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:15.163873    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9m7fb for pod openshift-network-diagnostics/network-check-target-dh95c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:15.165928 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:15.163941    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb podName:e9fd18af-03b0-434f-9ef3-a9b2775d545f nodeName:}" failed. No retries permitted until 2026-07-07 14:09:19.163921184 +0000 UTC m=+10.161511893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m7fb" (UniqueName: "kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb") pod "network-check-target-dh95c" (UID: "e9fd18af-03b0-434f-9ef3-a9b2775d545f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:15.483159 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:15.482886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:15.483159 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:15.483026    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:16.480822 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:16.480783    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:16.481371 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:16.480921    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:16.481371 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:16.481307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:16.481541 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:16.481397    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:17.481318 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:17.481113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:17.481318 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:17.481267    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:18.481151 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:18.481108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:18.481342 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:18.481243    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:18.481342 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:18.481311    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:18.481784 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:18.481378    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:19.096847 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:19.096804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:19.097034 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:19.096916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:19.097034 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:19.096960    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:19.097034 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:19.097034    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret podName:e8ae7e96-9ae1-45b4-a74a-fb810990dd5c nodeName:}" failed. No retries permitted until 2026-07-07 14:09:27.097017768 +0000 UTC m=+18.094608464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret") pod "global-pull-secret-syncer-m6ghz" (UID: "e8ae7e96-9ae1-45b4-a74a-fb810990dd5c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:19.097192 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:19.097046    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:19.097192 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:19.097101    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:27.097082739 +0000 UTC m=+18.094673450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:19.197680 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:19.197251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:19.197680 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:19.197471    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:19.197680 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:19.197489    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:19.197680 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:19.197502    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9m7fb for pod openshift-network-diagnostics/network-check-target-dh95c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:19.197680 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:19.197560    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb podName:e9fd18af-03b0-434f-9ef3-a9b2775d545f nodeName:}" failed. No retries permitted until 2026-07-07 14:09:27.197542753 +0000 UTC m=+18.195133446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m7fb" (UniqueName: "kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb") pod "network-check-target-dh95c" (UID: "e9fd18af-03b0-434f-9ef3-a9b2775d545f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:19.481866 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:19.481829    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:19.482280 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:19.481966    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:20.480910 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:20.480875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:20.481106 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:20.480887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:20.481106 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:20.481014    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:20.481106 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:20.481039    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:21.481539 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:21.481508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:21.482113 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:21.481678    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:22.481065 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:22.481031    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:22.481227 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:22.481125    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:22.481227 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:22.481184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:22.481357 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:22.481289    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:23.481010 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:23.480973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:23.481494 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:23.481120    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:24.480570 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:24.480533    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:24.480789 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:24.480535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:24.480789 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:24.480688    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:24.480789 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:24.480741    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:25.481526 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:25.481489    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:25.481947 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:25.481649    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:26.481485 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:26.481445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:26.481485 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:26.481476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:26.482060 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:26.481577    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:26.482060 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:26.481722    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:27.156484 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:27.156445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:27.156715 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:27.156530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:27.156715 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:27.156603    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:27.156715 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:27.156647    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:27.156715 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:27.156699    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret podName:e8ae7e96-9ae1-45b4-a74a-fb810990dd5c nodeName:}" failed. No retries permitted until 2026-07-07 14:09:43.15668005 +0000 UTC m=+34.154270759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret") pod "global-pull-secret-syncer-m6ghz" (UID: "e8ae7e96-9ae1-45b4-a74a-fb810990dd5c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:27.156715 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:27.156716    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:43.156708654 +0000 UTC m=+34.154299350 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:27.257243 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:27.257209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:27.257410 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:27.257394    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:27.257486 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:27.257416    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:27.257486 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:27.257428    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9m7fb for pod openshift-network-diagnostics/network-check-target-dh95c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:27.257563 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:27.257495    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb podName:e9fd18af-03b0-434f-9ef3-a9b2775d545f nodeName:}" failed. No retries permitted until 2026-07-07 14:09:43.257475618 +0000 UTC m=+34.255066330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m7fb" (UniqueName: "kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb") pod "network-check-target-dh95c" (UID: "e9fd18af-03b0-434f-9ef3-a9b2775d545f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:27.481006 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:27.480918    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:27.481169 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:27.481055    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:28.480769 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:28.480730    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:28.480769 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:28.480762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:28.481281 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:28.480871    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:28.481281 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:28.480975    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:29.482220 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:29.482187    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:29.482672 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:29.482293    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:30.481004 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:30.480962    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:30.481183 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:30.480985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:30.481183 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:30.481108    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:30.481183 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:30.481174    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:31.481363 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.481033    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:31.481979 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:31.481456    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:31.594515 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.594482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" event={"ID":"7b291786-af74-43e3-84e4-2af9581a134f","Type":"ContainerStarted","Data":"856936754f803a2f9ec645d02088af8ba9886bde8e069d7d9b40323e2bd4da25"}
Jul 07 14:09:31.594699 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.594557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" event={"ID":"7b291786-af74-43e3-84e4-2af9581a134f","Type":"ContainerStarted","Data":"79c21dbfecc5ac290fb40aa9cac44e913732e5f3da8c05c993f67f3fded038b5"}
Jul 07 14:09:31.594699 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.594603    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" event={"ID":"7b291786-af74-43e3-84e4-2af9581a134f","Type":"ContainerStarted","Data":"b88fb74669e5381aef041ffac0209bd90ffeb9838fcca160e4bc444165e257aa"}
Jul 07 14:09:31.595826 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.595798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xtl8q" event={"ID":"46a39168-6bed-493d-a1e0-c4313508adc4","Type":"ContainerStarted","Data":"25470fa50c375b422dc858a31394f7a0189367264057d06e984df642b2cee984"}
Jul 07 14:09:31.597135 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.597113    2565 generic.go:356] "Generic (PLEG): container finished" podID="c78112d6-bdd7-44c5-9214-04e37f5c16a2" containerID="6d7a3bf463b0cb9d9b273c557d6e1b04bdf13f227aa03fed84975a7cc164f8ef" exitCode=0
Jul 07 14:09:31.597228 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.597184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54wn6" event={"ID":"c78112d6-bdd7-44c5-9214-04e37f5c16a2","Type":"ContainerDied","Data":"6d7a3bf463b0cb9d9b273c557d6e1b04bdf13f227aa03fed84975a7cc164f8ef"}
Jul 07 14:09:31.598464 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.598412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p66vc" event={"ID":"b0b57686-10e6-4348-b196-2a49ace4083f","Type":"ContainerStarted","Data":"6d1a7c96d095a3f6e3ff4a235026b1890217519224eb85dc03e82c409cda6439"}
Jul 07 14:09:31.599922 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.599901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gj9qr" event={"ID":"d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed","Type":"ContainerStarted","Data":"2541a133b21b842f397ea1e6ed8b69d6e4ea14e7976e98704b68162509fbae28"}
Jul 07 14:09:31.601281 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.601262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45" event={"ID":"0ae7a068-2e12-4564-b34f-e8eee03f70bd","Type":"ContainerStarted","Data":"5e14a133be9c8817c9146a4bc08bc1aec521a045a4d1dc495c9fc03553dded36"}
Jul 07 14:09:31.602484 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.602466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kdp5h" event={"ID":"177c9d3c-7f21-498a-b90d-1c818b61a216","Type":"ContainerStarted","Data":"beb4c123a8e3d91b81375759489e83731654ab8abecc8348d53c98d8fa1334b7"}
Jul 07 14:09:31.616432 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.616385    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xtl8q" podStartSLOduration=3.288866522 podStartE2EDuration="22.616370352s" totalImagesPullingTime="19.32750383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:31.616208946 +0000 UTC m=+22.613799661" watchObservedRunningTime="2026-07-07 14:09:31.616370352 +0000 UTC m=+22.613961101"
Jul 07 14:09:31.616703 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.616592    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-82.ec2.internal" podStartSLOduration=21.616583407 podStartE2EDuration="21.616583407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:14.622284647 +0000 UTC m=+5.619875361" watchObservedRunningTime="2026-07-07 14:09:31.616583407 +0000 UTC m=+22.614174123"
Jul 07 14:09:31.661301 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.661251    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p66vc" podStartSLOduration=3.336910275 podStartE2EDuration="22.661236402s" totalImagesPullingTime="19.324326127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:31.660777841 +0000 UTC m=+22.658368556" watchObservedRunningTime="2026-07-07 14:09:31.661236402 +0000 UTC m=+22.658827116"
Jul 07 14:09:31.680767 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.680684    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kdp5h" podStartSLOduration=3.362174971 podStartE2EDuration="22.680668267s" totalImagesPullingTime="19.318493296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:31.680046009 +0000 UTC m=+22.677636723" watchObservedRunningTime="2026-07-07 14:09:31.680668267 +0000 UTC m=+22.678258981"
Jul 07 14:09:31.698274 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:31.698230    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gj9qr" podStartSLOduration=3.38205694 podStartE2EDuration="22.698215992s" totalImagesPullingTime="19.316159052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:31.697677273 +0000 UTC m=+22.695267988" watchObservedRunningTime="2026-07-07 14:09:31.698215992 +0000 UTC m=+22.695806707"
Jul 07 14:09:32.481307 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:32.481237    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:32.481565 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:32.481341    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:32.481565 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:32.481396    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:32.481565 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:32.481482    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:32.608034 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:32.608006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" event={"ID":"7b291786-af74-43e3-84e4-2af9581a134f","Type":"ContainerStarted","Data":"5066939f1bd7a41b61b703d59af6529347545fd7e3ae983db3706c4258c22916"}
Jul 07 14:09:32.608171 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:32.608083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" event={"ID":"7b291786-af74-43e3-84e4-2af9581a134f","Type":"ContainerStarted","Data":"c412ca8aacaa47a5d1d70c004aba08ba0785261d53caead72326248362c5176c"}
Jul 07 14:09:32.608171 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:32.608117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" event={"ID":"7b291786-af74-43e3-84e4-2af9581a134f","Type":"ContainerStarted","Data":"974a65f19d8d8ad2425162d41513016cdd4baafb05f07627465d6730a555310d"}
Jul 07 14:09:32.609294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:32.609265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bhjlt" event={"ID":"4ee39c5e-114a-4756-b5e3-f68d0d72c49b","Type":"ContainerStarted","Data":"8f962f8db668bde15cca58155d16d6969d55b1d5624eef66071e917ea377c0cf"}
Jul 07 14:09:32.720818 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:32.720789    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 14:09:33.386989 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:33.386873    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:09:32.720807211Z","UUID":"a1f37e3a-5283-40ab-9b62-9d1fe5af04f3","Handler":null,"Name":"","Endpoint":""}
Jul 07 14:09:33.388832 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:33.388799    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 14:09:33.388965 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:33.388849    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 14:09:33.481486 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:33.481450    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:33.481677 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:33.481591    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:33.494141 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:33.494106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:33.613573 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:33.613531    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45" event={"ID":"0ae7a068-2e12-4564-b34f-e8eee03f70bd","Type":"ContainerStarted","Data":"79513022ee50179f728f1dcf722f632863748f2a096901e2cefc2aef5d745b56"}
Jul 07 14:09:33.630263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:33.630228    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:33.630982 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:33.630961    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:33.648899 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:33.648801    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bhjlt" podStartSLOduration=5.704370599 podStartE2EDuration="24.648779236s" totalImagesPullingTime="18.944408637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:32.635232525 +0000 UTC m=+23.632823240" watchObservedRunningTime="2026-07-07 14:09:33.648779236 +0000 UTC m=+24.646369954"
Jul 07 14:09:34.480958 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:34.480783    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:34.481135 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:34.480785    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:34.481135 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:34.481057    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:34.481135 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:34.481098    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:34.620228 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:34.620179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" event={"ID":"7b291786-af74-43e3-84e4-2af9581a134f","Type":"ContainerStarted","Data":"15624d435ab671a452ea3e37d428a50de74255badb364da728b61d4d489d5492"}
Jul 07 14:09:34.622475 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:34.622438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45" event={"ID":"0ae7a068-2e12-4564-b34f-e8eee03f70bd","Type":"ContainerStarted","Data":"16e4f5f29a08ac801b165906911d4a9b0041e5a82c82d9a3f5e165207e94bc01"}
Jul 07 14:09:34.623294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:34.623271    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p66vc"
Jul 07 14:09:34.657714 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:34.657576    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jjm45" podStartSLOduration=4.535113119 podStartE2EDuration="25.657552179s" totalImagesPullingTime="21.12243906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:34.642929907 +0000 UTC m=+25.640520625" watchObservedRunningTime="2026-07-07 14:09:34.657552179 +0000 UTC m=+25.655142895"
Jul 07 14:09:35.484404 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:35.484375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:35.484605 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:35.484502    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:36.481089 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:36.480997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:36.481799 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:36.481007    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:36.481799 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:36.481144    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:36.481799 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:36.481220    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:37.481506 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:37.481319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:37.481984 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:37.481592    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:37.631935 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:37.631898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" event={"ID":"7b291786-af74-43e3-84e4-2af9581a134f","Type":"ContainerStarted","Data":"92dafd88a52c504107ccd2a34bfa35673aa34c7c32d9dfdd83c3812261f50893"}
Jul 07 14:09:37.632145 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:37.632120    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:37.633681 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:37.633656    2565 generic.go:356] "Generic (PLEG): container finished" podID="c78112d6-bdd7-44c5-9214-04e37f5c16a2" containerID="bb4dbfd286b5362989ca49c1fb42429fe5ccc3a920d4416986678059391fc0f8" exitCode=0
Jul 07 14:09:37.633796 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:37.633698    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54wn6" event={"ID":"c78112d6-bdd7-44c5-9214-04e37f5c16a2","Type":"ContainerDied","Data":"bb4dbfd286b5362989ca49c1fb42429fe5ccc3a920d4416986678059391fc0f8"}
Jul 07 14:09:37.648403 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:37.648373    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:37.665997 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:37.665951    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw" podStartSLOduration=9.199088884 podStartE2EDuration="28.665936826s" totalImagesPullingTime="19.466847942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:37.665651074 +0000 UTC m=+28.663241816" watchObservedRunningTime="2026-07-07 14:09:37.665936826 +0000 UTC m=+28.663527558"
Jul 07 14:09:38.480687 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:38.480660    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:38.481126 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:38.480663    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:38.481126 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:38.480768    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:38.481126 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:38.480821    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:38.636429 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:38.636395    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:38.636977 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:38.636494    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:38.654409 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:38.654383    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:09:39.015336 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:39.015156    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dh95c"]
Jul 07 14:09:39.015496 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:39.015460    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:39.015577 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:39.015554    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:39.020192 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:39.020167    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m6ghz"]
Jul 07 14:09:39.020310 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:39.020270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:39.020363 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:39.020347    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:39.040797 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:39.040731    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-znqh7"]
Jul 07 14:09:39.040927 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:39.040848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:39.040969 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:39.040927    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:39.640566 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:39.640534    2565 generic.go:356] "Generic (PLEG): container finished" podID="c78112d6-bdd7-44c5-9214-04e37f5c16a2" containerID="d1e05cd2596391fc9eaab7149a602b830b3b0d0f754c3312fa93914a3ff951ec" exitCode=0
Jul 07 14:09:39.641141 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:39.640635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54wn6" event={"ID":"c78112d6-bdd7-44c5-9214-04e37f5c16a2","Type":"ContainerDied","Data":"d1e05cd2596391fc9eaab7149a602b830b3b0d0f754c3312fa93914a3ff951ec"}
Jul 07 14:09:40.480693 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:40.480659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:40.480896 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:40.480664    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:40.480896 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:40.480824    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:40.480896 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:40.480674    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:40.481238 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:40.481191    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:40.481388 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:40.481366    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:41.647375 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:41.647340    2565 generic.go:356] "Generic (PLEG): container finished" podID="c78112d6-bdd7-44c5-9214-04e37f5c16a2" containerID="a6c46a1d116026f292733839fded218dfebb1da773c293a1fd7fb2ea8b45efec" exitCode=0
Jul 07 14:09:41.648131 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:41.647401    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54wn6" event={"ID":"c78112d6-bdd7-44c5-9214-04e37f5c16a2","Type":"ContainerDied","Data":"a6c46a1d116026f292733839fded218dfebb1da773c293a1fd7fb2ea8b45efec"}
Jul 07 14:09:42.481408 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:42.481369    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:42.481408 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:42.481406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:42.481668 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:42.481421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:42.481668 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:42.481508    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:42.481760 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:42.481689    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:42.481797 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:42.481780    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:43.172505 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:43.172470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:43.172973 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:43.172523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:43.172973 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:43.172642    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:43.172973 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:43.172700    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret podName:e8ae7e96-9ae1-45b4-a74a-fb810990dd5c nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.172683837 +0000 UTC m=+66.170274530 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret") pod "global-pull-secret-syncer-m6ghz" (UID: "e8ae7e96-9ae1-45b4-a74a-fb810990dd5c") : object "kube-system"/"original-pull-secret" not registered
Jul 07 14:09:43.172973 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:43.172642    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:43.172973 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:43.172780    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.17275672 +0000 UTC m=+66.170347417 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 14:09:43.273255 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:43.273216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:43.273447 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:43.273423    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 14:09:43.273511 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:43.273458    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 14:09:43.273511 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:43.273475    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9m7fb for pod openshift-network-diagnostics/network-check-target-dh95c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:43.273606 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:43.273545    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb podName:e9fd18af-03b0-434f-9ef3-a9b2775d545f nodeName:}" failed. No retries permitted until 2026-07-07 14:10:15.273525783 +0000 UTC m=+66.271116480 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9m7fb" (UniqueName: "kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb") pod "network-check-target-dh95c" (UID: "e9fd18af-03b0-434f-9ef3-a9b2775d545f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 14:09:44.481165 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.481132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:44.481665 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.481145    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:44.481665 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:44.481257    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6ghz" podUID="e8ae7e96-9ae1-45b4-a74a-fb810990dd5c"
Jul 07 14:09:44.481665 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.481145    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:44.481665 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:44.481338    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:09:44.481665 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:44.481427    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dh95c" podUID="e9fd18af-03b0-434f-9ef3-a9b2775d545f"
Jul 07 14:09:44.852818 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.852507    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-82.ec2.internal" event="NodeReady"
Jul 07 14:09:44.852975 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.852867    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 14:09:44.946399 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.946364    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wkpv6"]
Jul 07 14:09:44.951427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.951397    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bbxsw"]
Jul 07 14:09:44.951633 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.951481    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wkpv6"
Jul 07 14:09:44.954155 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.954124    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 14:09:44.954292 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.954158    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ppbqm\""
Jul 07 14:09:44.954292 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.954263    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 14:09:44.954520 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.954503    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 14:09:44.955252 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.955221    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:44.957385 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.957364    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 14:09:44.957649 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.957632    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2225x\""
Jul 07 14:09:44.957649 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.957641    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 14:09:44.965834 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.965661    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wkpv6"]
Jul 07 14:09:44.966581 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:44.966556    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bbxsw"]
Jul 07 14:09:45.084966 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.084917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c200e335-efdd-4d8b-bb88-42fa4bfecf13-cert\") pod \"ingress-canary-wkpv6\" (UID: \"c200e335-efdd-4d8b-bb88-42fa4bfecf13\") " pod="openshift-ingress-canary/ingress-canary-wkpv6"
Jul 07 14:09:45.084966 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.084970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc20856-2dc3-409d-aa1c-92eb8ff59349-config-volume\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.085206 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.085001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.085206 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.085052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc20856-2dc3-409d-aa1c-92eb8ff59349-tmp-dir\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.085206 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.085083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4h7c\" (UniqueName: \"kubernetes.io/projected/c200e335-efdd-4d8b-bb88-42fa4bfecf13-kube-api-access-m4h7c\") pod \"ingress-canary-wkpv6\" (UID: \"c200e335-efdd-4d8b-bb88-42fa4bfecf13\") " pod="openshift-ingress-canary/ingress-canary-wkpv6"
Jul 07 14:09:45.085206 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.085148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5cl\" (UniqueName: \"kubernetes.io/projected/7bc20856-2dc3-409d-aa1c-92eb8ff59349-kube-api-access-lf5cl\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.185821 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.185772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5cl\" (UniqueName: \"kubernetes.io/projected/7bc20856-2dc3-409d-aa1c-92eb8ff59349-kube-api-access-lf5cl\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.186021 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.185853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c200e335-efdd-4d8b-bb88-42fa4bfecf13-cert\") pod \"ingress-canary-wkpv6\" (UID: \"c200e335-efdd-4d8b-bb88-42fa4bfecf13\") " pod="openshift-ingress-canary/ingress-canary-wkpv6"
Jul 07 14:09:45.186021 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.185886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc20856-2dc3-409d-aa1c-92eb8ff59349-config-volume\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.186021 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.185911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.186021 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.185943    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc20856-2dc3-409d-aa1c-92eb8ff59349-tmp-dir\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.186021 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.185966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4h7c\" (UniqueName: \"kubernetes.io/projected/c200e335-efdd-4d8b-bb88-42fa4bfecf13-kube-api-access-m4h7c\") pod \"ingress-canary-wkpv6\" (UID: \"c200e335-efdd-4d8b-bb88-42fa4bfecf13\") " pod="openshift-ingress-canary/ingress-canary-wkpv6"
Jul 07 14:09:45.186273 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:45.186204    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:45.186326 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:45.186278    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls podName:7bc20856-2dc3-409d-aa1c-92eb8ff59349 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:45.686260088 +0000 UTC m=+36.683850795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls") pod "dns-default-bbxsw" (UID: "7bc20856-2dc3-409d-aa1c-92eb8ff59349") : secret "dns-default-metrics-tls" not found
Jul 07 14:09:45.186393 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.186359    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc20856-2dc3-409d-aa1c-92eb8ff59349-tmp-dir\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.186609 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.186580    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc20856-2dc3-409d-aa1c-92eb8ff59349-config-volume\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.190687 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.190669    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c200e335-efdd-4d8b-bb88-42fa4bfecf13-cert\") pod \"ingress-canary-wkpv6\" (UID: \"c200e335-efdd-4d8b-bb88-42fa4bfecf13\") " pod="openshift-ingress-canary/ingress-canary-wkpv6"
Jul 07 14:09:45.195410 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.195385    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5cl\" (UniqueName: \"kubernetes.io/projected/7bc20856-2dc3-409d-aa1c-92eb8ff59349-kube-api-access-lf5cl\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.197990 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.197970    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4h7c\" (UniqueName: \"kubernetes.io/projected/c200e335-efdd-4d8b-bb88-42fa4bfecf13-kube-api-access-m4h7c\") pod \"ingress-canary-wkpv6\" (UID: \"c200e335-efdd-4d8b-bb88-42fa4bfecf13\") " pod="openshift-ingress-canary/ingress-canary-wkpv6"
Jul 07 14:09:45.264097 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.264063    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wkpv6"
Jul 07 14:09:45.691130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.691096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:45.691552 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:45.691260    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:45.691552 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:45.691343    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls podName:7bc20856-2dc3-409d-aa1c-92eb8ff59349 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:46.691320176 +0000 UTC m=+37.688910890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls") pod "dns-default-bbxsw" (UID: "7bc20856-2dc3-409d-aa1c-92eb8ff59349") : secret "dns-default-metrics-tls" not found
Jul 07 14:09:45.899936 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:45.899905    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wkpv6"]
Jul 07 14:09:45.907712 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:09:45.907687    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc200e335_efdd_4d8b_bb88_42fa4bfecf13.slice/crio-b28964a583345c50453a0c9dbe601f2bd8a453578a1354242a8803c27e861dbb WatchSource:0}: Error finding container b28964a583345c50453a0c9dbe601f2bd8a453578a1354242a8803c27e861dbb: Status 404 returned error can't find the container with id b28964a583345c50453a0c9dbe601f2bd8a453578a1354242a8803c27e861dbb
Jul 07 14:09:46.481066 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.481024    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:09:46.481240 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.481044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:09:46.481240 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.481164    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:09:46.483910 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.483707    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:09:46.484033 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.483910    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lp7v4\""
Jul 07 14:09:46.484719 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.484697    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:09:46.484832 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.484800    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:09:46.484961 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.484942    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:09:46.485039 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.485011    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2hv7\""
Jul 07 14:09:46.660926 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.660882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wkpv6" event={"ID":"c200e335-efdd-4d8b-bb88-42fa4bfecf13","Type":"ContainerStarted","Data":"b28964a583345c50453a0c9dbe601f2bd8a453578a1354242a8803c27e861dbb"}
Jul 07 14:09:46.664050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.664018    2565 generic.go:356] "Generic (PLEG): container finished" podID="c78112d6-bdd7-44c5-9214-04e37f5c16a2" containerID="1c4b4895de94b958fc830a59a584d45602a99eb40c2b00afe83d740cc21ef534" exitCode=0
Jul 07 14:09:46.664214 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.664068    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54wn6" event={"ID":"c78112d6-bdd7-44c5-9214-04e37f5c16a2","Type":"ContainerDied","Data":"1c4b4895de94b958fc830a59a584d45602a99eb40c2b00afe83d740cc21ef534"}
Jul 07 14:09:46.698761 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:46.698727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:46.699162 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:46.698928    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:46.699162 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:46.699038    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls podName:7bc20856-2dc3-409d-aa1c-92eb8ff59349 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:48.699005432 +0000 UTC m=+39.696596128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls") pod "dns-default-bbxsw" (UID: "7bc20856-2dc3-409d-aa1c-92eb8ff59349") : secret "dns-default-metrics-tls" not found
Jul 07 14:09:47.671452 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:47.671201    2565 generic.go:356] "Generic (PLEG): container finished" podID="c78112d6-bdd7-44c5-9214-04e37f5c16a2" containerID="2c13b3178874f1d8e58ddda44e64b12d1e20172f93c275a62ab0ec7f6f02fab7" exitCode=0
Jul 07 14:09:47.671452 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:47.671253    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54wn6" event={"ID":"c78112d6-bdd7-44c5-9214-04e37f5c16a2","Type":"ContainerDied","Data":"2c13b3178874f1d8e58ddda44e64b12d1e20172f93c275a62ab0ec7f6f02fab7"}
Jul 07 14:09:48.676078 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:48.675970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wkpv6" event={"ID":"c200e335-efdd-4d8b-bb88-42fa4bfecf13","Type":"ContainerStarted","Data":"1c049c8aa70f746cd720d02eea83f665a6b7da0305075ba54360ff67c292474b"}
Jul 07 14:09:48.679449 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:48.679422    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54wn6" event={"ID":"c78112d6-bdd7-44c5-9214-04e37f5c16a2","Type":"ContainerStarted","Data":"681fe4fe5d38d3b9fbfb36c5bb57282ea795159e7222ad7e6e79c48978501307"}
Jul 07 14:09:48.695968 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:48.695917    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wkpv6" podStartSLOduration=2.159522296 podStartE2EDuration="4.695900488s" totalImagesPullingTime="2.536378192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:48.694732125 +0000 UTC m=+39.692322840" watchObservedRunningTime="2026-07-07 14:09:48.695900488 +0000 UTC m=+39.693491203"
Jul 07 14:09:48.714303 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:48.714272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:48.714483 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:48.714417    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:48.714538 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:48.714483    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls podName:7bc20856-2dc3-409d-aa1c-92eb8ff59349 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:52.714466715 +0000 UTC m=+43.712057412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls") pod "dns-default-bbxsw" (UID: "7bc20856-2dc3-409d-aa1c-92eb8ff59349") : secret "dns-default-metrics-tls" not found
Jul 07 14:09:52.741332 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:09:52.741290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:09:52.741769 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:52.741439    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:09:52.741769 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:09:52.741506    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls podName:7bc20856-2dc3-409d-aa1c-92eb8ff59349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:00.741488568 +0000 UTC m=+51.739079269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls") pod "dns-default-bbxsw" (UID: "7bc20856-2dc3-409d-aa1c-92eb8ff59349") : secret "dns-default-metrics-tls" not found
Jul 07 14:10:00.799241 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:00.799192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:10:00.799740 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:10:00.799350    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:10:00.799740 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:10:00.799426    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls podName:7bc20856-2dc3-409d-aa1c-92eb8ff59349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:16.799408176 +0000 UTC m=+67.796998868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls") pod "dns-default-bbxsw" (UID: "7bc20856-2dc3-409d-aa1c-92eb8ff59349") : secret "dns-default-metrics-tls" not found
Jul 07 14:10:10.653272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:10.653244    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-82xpw"
Jul 07 14:10:10.685567 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:10.685518    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-54wn6" podStartSLOduration=30.133635753 podStartE2EDuration="1m1.685502014s" totalImagesPullingTime="31.551866261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:09:48.721807251 +0000 UTC m=+39.719397965" watchObservedRunningTime="2026-07-07 14:10:10.685502014 +0000 UTC m=+61.683092730"
Jul 07 14:10:15.208593 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.208542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:10:15.209087 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.208633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:10:15.211679 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.211658    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 14:10:15.211779 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.211661    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 14:10:15.218955 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:10:15.218933    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 14:10:15.219007 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:10:15.218997    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:19.218980807 +0000 UTC m=+130.216571513 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : secret "metrics-daemon-secret" not found
Jul 07 14:10:15.222236 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.222214    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e8ae7e96-9ae1-45b4-a74a-fb810990dd5c-original-pull-secret\") pod \"global-pull-secret-syncer-m6ghz\" (UID: \"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c\") " pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:10:15.294714 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.294677    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6ghz"
Jul 07 14:10:15.309526 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.309486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:10:15.312331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.312311    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 14:10:15.322755 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.322726    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 14:10:15.334037 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.333316    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7fb\" (UniqueName: \"kubernetes.io/projected/e9fd18af-03b0-434f-9ef3-a9b2775d545f-kube-api-access-9m7fb\") pod \"network-check-target-dh95c\" (UID: \"e9fd18af-03b0-434f-9ef3-a9b2775d545f\") " pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:10:15.424131 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.424092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m6ghz"]
Jul 07 14:10:15.427466 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:10:15.427438    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ae7e96_9ae1_45b4_a74a_fb810990dd5c.slice/crio-b198aea2c16099971b9bb2bc4127a178bc372dec37050e5ad07593e79c1f11c9 WatchSource:0}: Error finding container b198aea2c16099971b9bb2bc4127a178bc372dec37050e5ad07593e79c1f11c9: Status 404 returned error can't find the container with id b198aea2c16099971b9bb2bc4127a178bc372dec37050e5ad07593e79c1f11c9
Jul 07 14:10:15.603685 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.603594    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lp7v4\""
Jul 07 14:10:15.611333 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.611308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:10:15.738763 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.738726    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dh95c"]
Jul 07 14:10:15.741817 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:10:15.741789    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fd18af_03b0_434f_9ef3_a9b2775d545f.slice/crio-20d9c89dd8ddf260d2cce7a027dfd1f435603be8d3018963b2067a650f11ee5b WatchSource:0}: Error finding container 20d9c89dd8ddf260d2cce7a027dfd1f435603be8d3018963b2067a650f11ee5b: Status 404 returned error can't find the container with id 20d9c89dd8ddf260d2cce7a027dfd1f435603be8d3018963b2067a650f11ee5b
Jul 07 14:10:15.743746 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:15.743715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m6ghz" event={"ID":"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c","Type":"ContainerStarted","Data":"b198aea2c16099971b9bb2bc4127a178bc372dec37050e5ad07593e79c1f11c9"}
Jul 07 14:10:16.748318 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:16.748282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dh95c" event={"ID":"e9fd18af-03b0-434f-9ef3-a9b2775d545f","Type":"ContainerStarted","Data":"20d9c89dd8ddf260d2cce7a027dfd1f435603be8d3018963b2067a650f11ee5b"}
Jul 07 14:10:16.820180 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:16.820143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:10:16.820384 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:10:16.820278    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:10:16.820384 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:10:16.820350    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls podName:7bc20856-2dc3-409d-aa1c-92eb8ff59349 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:48.820328492 +0000 UTC m=+99.817919189 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls") pod "dns-default-bbxsw" (UID: "7bc20856-2dc3-409d-aa1c-92eb8ff59349") : secret "dns-default-metrics-tls" not found
Jul 07 14:10:20.761240 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:20.761195    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m6ghz" event={"ID":"e8ae7e96-9ae1-45b4-a74a-fb810990dd5c","Type":"ContainerStarted","Data":"030e0a829c8c61c65603f88c13c824b3ef352312251d3aaf814de698afbd8792"}
Jul 07 14:10:20.762594 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:20.762572    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dh95c" event={"ID":"e9fd18af-03b0-434f-9ef3-a9b2775d545f","Type":"ContainerStarted","Data":"5c66c9821b68cc120b4c42bca725d50b45038c27ef2554636cda2ce5031e8d74"}
Jul 07 14:10:20.762705 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:20.762651    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:10:20.826602 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:20.826540    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m6ghz" podStartSLOduration=66.909208432 podStartE2EDuration="1m11.826522214s" totalImagesPullingTime="4.917313782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:20.794899707 +0000 UTC m=+71.792490422" watchObservedRunningTime="2026-07-07 14:10:20.826522214 +0000 UTC m=+71.824112975"
Jul 07 14:10:20.826795 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:20.826712    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dh95c" podStartSLOduration=67.223757397 podStartE2EDuration="1m11.826706576s" totalImagesPullingTime="4.602949179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:20.826121055 +0000 UTC m=+71.823711782" watchObservedRunningTime="2026-07-07 14:10:20.826706576 +0000 UTC m=+71.824297290"
Jul 07 14:10:25.617582 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.617543    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"]
Jul 07 14:10:25.621463 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.621442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.624963 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.624935    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 14:10:25.625099 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.624940    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 14:10:25.625099 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.624939    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 14:10:25.625099 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.624972    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 14:10:25.638071 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.638041    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"]
Jul 07 14:10:25.663335 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.663304    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"]
Jul 07 14:10:25.667367 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.667351    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.670771 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.670750    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 07 14:10:25.670888 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.670752    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 07 14:10:25.670888 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.670752    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 07 14:10:25.671063 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.671044    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 07 14:10:25.682234 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.682212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/74b6071a-e5c6-4701-b907-941ab8c1b51e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f55847d46-j2xh4\" (UID: \"74b6071a-e5c6-4701-b907-941ab8c1b51e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.682357 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.682247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbhn\" (UniqueName: \"kubernetes.io/projected/74b6071a-e5c6-4701-b907-941ab8c1b51e-kube-api-access-twbhn\") pod \"klusterlet-addon-workmgr-6f55847d46-j2xh4\" (UID: \"74b6071a-e5c6-4701-b907-941ab8c1b51e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.682357 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.682282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74b6071a-e5c6-4701-b907-941ab8c1b51e-tmp\") pod \"klusterlet-addon-workmgr-6f55847d46-j2xh4\" (UID: \"74b6071a-e5c6-4701-b907-941ab8c1b51e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.684118 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.684094    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"]
Jul 07 14:10:25.782951 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.782917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.783132 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.782977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74b6071a-e5c6-4701-b907-941ab8c1b51e-tmp\") pod \"klusterlet-addon-workmgr-6f55847d46-j2xh4\" (UID: \"74b6071a-e5c6-4701-b907-941ab8c1b51e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.783132 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.783045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.783132 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.783080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlmsf\" (UniqueName: \"kubernetes.io/projected/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-kube-api-access-dlmsf\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.783132 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.783120    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-hub\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.783294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.783149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/74b6071a-e5c6-4701-b907-941ab8c1b51e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f55847d46-j2xh4\" (UID: \"74b6071a-e5c6-4701-b907-941ab8c1b51e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.783294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.783187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.783294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.783232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbhn\" (UniqueName: \"kubernetes.io/projected/74b6071a-e5c6-4701-b907-941ab8c1b51e-kube-api-access-twbhn\") pod \"klusterlet-addon-workmgr-6f55847d46-j2xh4\" (UID: \"74b6071a-e5c6-4701-b907-941ab8c1b51e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.783294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.783267    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-ca\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.783420 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.783389    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74b6071a-e5c6-4701-b907-941ab8c1b51e-tmp\") pod \"klusterlet-addon-workmgr-6f55847d46-j2xh4\" (UID: \"74b6071a-e5c6-4701-b907-941ab8c1b51e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.785611 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.785588    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/74b6071a-e5c6-4701-b907-941ab8c1b51e-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f55847d46-j2xh4\" (UID: \"74b6071a-e5c6-4701-b907-941ab8c1b51e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.795706 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.795680    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbhn\" (UniqueName: \"kubernetes.io/projected/74b6071a-e5c6-4701-b907-941ab8c1b51e-kube-api-access-twbhn\") pod \"klusterlet-addon-workmgr-6f55847d46-j2xh4\" (UID: \"74b6071a-e5c6-4701-b907-941ab8c1b51e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.883942 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.883897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.883942 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.883948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlmsf\" (UniqueName: \"kubernetes.io/projected/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-kube-api-access-dlmsf\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.884130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.884076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-hub\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.884130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.884114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.884219 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.884162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-ca\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.884219 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.884198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.884704 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.884678    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.886693 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.886667    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.886803 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.886677    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-hub\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.886866 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.886850    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-ca\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.887003 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.886985    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.896086 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.896051    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlmsf\" (UniqueName: \"kubernetes.io/projected/d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0-kube-api-access-dlmsf\") pod \"cluster-proxy-proxy-agent-58b9cd9f69-m6d77\" (UID: \"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:25.930788 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.930751    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:25.990704 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:25.990666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:10:26.065436 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:26.065399    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"]
Jul 07 14:10:26.069959 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:10:26.069925    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b6071a_e5c6_4701_b907_941ab8c1b51e.slice/crio-74c8ea9fef25a9f66de32df84dc5b296b2565e7a0c9b0b6ba4c8a45bc3e4135d WatchSource:0}: Error finding container 74c8ea9fef25a9f66de32df84dc5b296b2565e7a0c9b0b6ba4c8a45bc3e4135d: Status 404 returned error can't find the container with id 74c8ea9fef25a9f66de32df84dc5b296b2565e7a0c9b0b6ba4c8a45bc3e4135d
Jul 07 14:10:26.129582 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:26.129553    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"]
Jul 07 14:10:26.135027 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:10:26.134975    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6f4b43c_62c2_40a9_b977_7bcd6a17f5b0.slice/crio-fa42a135b2714dd28722c046841e48784bd712504796f4a03ae2ed5366d5657e WatchSource:0}: Error finding container fa42a135b2714dd28722c046841e48784bd712504796f4a03ae2ed5366d5657e: Status 404 returned error can't find the container with id fa42a135b2714dd28722c046841e48784bd712504796f4a03ae2ed5366d5657e
Jul 07 14:10:26.778006 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:26.777963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" event={"ID":"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0","Type":"ContainerStarted","Data":"fa42a135b2714dd28722c046841e48784bd712504796f4a03ae2ed5366d5657e"}
Jul 07 14:10:26.779016 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:26.778994    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4" event={"ID":"74b6071a-e5c6-4701-b907-941ab8c1b51e","Type":"ContainerStarted","Data":"74c8ea9fef25a9f66de32df84dc5b296b2565e7a0c9b0b6ba4c8a45bc3e4135d"}
Jul 07 14:10:31.802167 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:31.802078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" event={"ID":"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0","Type":"ContainerStarted","Data":"7e7e79581dfb1aa744eb120b37da7b9505882c6c830d2dc5ea1fc4f6009958e1"}
Jul 07 14:10:31.802167 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:31.802129    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" event={"ID":"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0","Type":"ContainerStarted","Data":"a5e47a10f3ebcbb3e55ad101f5e84e7caf0ce58458825c3d12080195db6b5b5f"}
Jul 07 14:10:31.802167 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:31.802142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" event={"ID":"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0","Type":"ContainerStarted","Data":"793f3c7bb2226fb2f07ea3ac9bea33c9acc486ad651b5923c777de95dfac55dc"}
Jul 07 14:10:31.803514 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:31.803486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4" event={"ID":"74b6071a-e5c6-4701-b907-941ab8c1b51e","Type":"ContainerStarted","Data":"7851c5ab5e3c5b3e89dfd640271d2610658238ff033fa0cfa18cb7387686558e"}
Jul 07 14:10:31.803648 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:31.803570    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:31.805299 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:31.805279    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:10:31.840263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:31.840210    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" podStartSLOduration=1.598274869 podStartE2EDuration="6.840195087s" totalImagesPullingTime="5.241920218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:31.839915063 +0000 UTC m=+82.837505789" watchObservedRunningTime="2026-07-07 14:10:31.840195087 +0000 UTC m=+82.837785803"
Jul 07 14:10:31.877718 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:31.877664    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4" podStartSLOduration=1.5705124289999999 podStartE2EDuration="6.877646413s" totalImagesPullingTime="5.307133984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:31.876587811 +0000 UTC m=+82.874178540" watchObservedRunningTime="2026-07-07 14:10:31.877646413 +0000 UTC m=+82.875237125"
Jul 07 14:10:48.846361 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:48.846253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:10:48.846816 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:10:48.846371    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:10:48.846816 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:10:48.846451    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls podName:7bc20856-2dc3-409d-aa1c-92eb8ff59349 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:52.846433581 +0000 UTC m=+163.844024274 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls") pod "dns-default-bbxsw" (UID: "7bc20856-2dc3-409d-aa1c-92eb8ff59349") : secret "dns-default-metrics-tls" not found
Jul 07 14:10:51.767015 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:10:51.766985    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dh95c"
Jul 07 14:11:19.262262 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:11:19.262210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:11:19.262888 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:11:19.262358    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 14:11:19.262888 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:11:19.262440    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:13:21.262423173 +0000 UTC m=+252.260013867 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : secret "metrics-daemon-secret" not found
Jul 07 14:11:47.971236 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:11:47.971181    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bbxsw" podUID="7bc20856-2dc3-409d-aa1c-92eb8ff59349"
Jul 07 14:11:48.020590 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:11:48.020564    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:11:49.513638 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:11:49.513560    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:11:52.901688 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:11:52.901652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:11:52.902085 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:11:52.901816    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 14:11:52.902085 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:11:52.901897    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls podName:7bc20856-2dc3-409d-aa1c-92eb8ff59349 nodeName:}" failed. No retries permitted until 2026-07-07 14:13:54.901876517 +0000 UTC m=+285.899467227 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls") pod "dns-default-bbxsw" (UID: "7bc20856-2dc3-409d-aa1c-92eb8ff59349") : secret "dns-default-metrics-tls" not found
Jul 07 14:12:00.480890 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:12:00.480850    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:12:31.804188 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:12:31.804089    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4" podUID="74b6071a-e5c6-4701-b907-941ab8c1b51e" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused"
Jul 07 14:12:32.147744 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:12:32.147711    2565 generic.go:356] "Generic (PLEG): container finished" podID="74b6071a-e5c6-4701-b907-941ab8c1b51e" containerID="7851c5ab5e3c5b3e89dfd640271d2610658238ff033fa0cfa18cb7387686558e" exitCode=1
Jul 07 14:12:32.147942 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:12:32.147778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4" event={"ID":"74b6071a-e5c6-4701-b907-941ab8c1b51e","Type":"ContainerDied","Data":"7851c5ab5e3c5b3e89dfd640271d2610658238ff033fa0cfa18cb7387686558e"}
Jul 07 14:12:32.148153 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:12:32.148137    2565 scope.go:117] "RemoveContainer" containerID="7851c5ab5e3c5b3e89dfd640271d2610658238ff033fa0cfa18cb7387686558e"
Jul 07 14:12:33.152693 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:12:33.152652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4" event={"ID":"74b6071a-e5c6-4701-b907-941ab8c1b51e","Type":"ContainerStarted","Data":"0854f6c2a04967785263922e346b821fcb829d6b8647a06c2ce700675cfb9d9f"}
Jul 07 14:12:33.153069 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:12:33.152955    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:12:33.153584 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:12:33.153569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f55847d46-j2xh4"
Jul 07 14:13:21.328890 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:21.328839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:13:21.329372 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:21.329004    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 14:13:21.329372 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:21.329066    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs podName:a92bcc45-e37c-46ed-9887-6a895f7e7104 nodeName:}" failed. No retries permitted until 2026-07-07 14:15:23.329050151 +0000 UTC m=+374.326640844 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs") pod "network-metrics-daemon-znqh7" (UID: "a92bcc45-e37c-46ed-9887-6a895f7e7104") : secret "metrics-daemon-secret" not found
Jul 07 14:13:42.902958 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.902868    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dq65l"]
Jul 07 14:13:42.905969 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.905937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:42.908420 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.908395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 14:13:42.909324 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.909305    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 14:13:42.909407 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.909320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xcd9z\""
Jul 07 14:13:42.909407 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.909337    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 14:13:42.909407 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.909336    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 14:13:42.918317 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.918294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dq65l"]
Jul 07 14:13:42.971882 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.971844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-data-volume\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:42.971882 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.971886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:42.972139 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.971920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcz7\" (UniqueName: \"kubernetes.io/projected/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-api-access-tjcz7\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:42.972139 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.971990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-crio-socket\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:42.972139 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:42.972037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.073033 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.072985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-data-volume\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.073033 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.073034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.073291 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.073065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcz7\" (UniqueName: \"kubernetes.io/projected/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-api-access-tjcz7\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.073291 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.073108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-crio-socket\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.073291 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.073175    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-crio-socket\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.073291 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:43.073176    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:43.073291 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.073223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.073291 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:43.073246    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:43.573228703 +0000 UTC m=+274.570819396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:43.073291 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:43.073270    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:13:43.073792 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.073613    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-data-volume\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.073792 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:43.073676    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:43.57365497 +0000 UTC m=+274.571245680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : configmap "kube-rbac-proxy" not found
Jul 07 14:13:43.083172 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.083145    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcz7\" (UniqueName: \"kubernetes.io/projected/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-api-access-tjcz7\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.578041 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.578002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.578209 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:43.578055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:43.578209 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:43.578126    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:13:43.578209 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:43.578183    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:43.578209 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:43.578193    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:44.578176565 +0000 UTC m=+275.575767261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : configmap "kube-rbac-proxy" not found
Jul 07 14:13:43.578346 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:43.578232    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:44.578220308 +0000 UTC m=+275.575811002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:44.586849 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:44.586810    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:44.587214 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:44.586877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:44.587214 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:44.586947    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:13:44.587214 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:44.586963    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:44.587214 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:44.587000    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:46.586984446 +0000 UTC m=+277.584575139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : configmap "kube-rbac-proxy" not found
Jul 07 14:13:44.587214 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:44.587026    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:46.587013228 +0000 UTC m=+277.584603921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:45.868591 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:45.868560    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gj9qr_d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed/dns-node-resolver/0.log"
Jul 07 14:13:46.603582 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:46.603531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:46.603826 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:46.603603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:46.603826 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:46.603686    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:13:46.603826 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:46.603743    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:46.603826 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:46.603786    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:50.603768695 +0000 UTC m=+281.601359388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : configmap "kube-rbac-proxy" not found
Jul 07 14:13:46.603826 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:46.603819    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:50.603802378 +0000 UTC m=+281.601393077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:46.668991 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:46.668967    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9xdbg_fd47c6c1-367e-416b-8c25-b64c869d5c22/node-ca/0.log"
Jul 07 14:13:47.471142 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:47.471113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wkpv6_c200e335-efdd-4d8b-bb88-42fa4bfecf13/serve-healthcheck-canary/0.log"
Jul 07 14:13:50.633413 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:50.633374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:50.633906 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:50.633435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:50.633906 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:50.633495    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 14:13:50.633906 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:50.633525    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:50.633906 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:50.633574    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:58.633560778 +0000 UTC m=+289.631151471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : secret "insights-runtime-extractor-tls" not found
Jul 07 14:13:50.633906 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:50.633587    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm podName:e9b9bfcb-3ffe-4b06-bdea-a0be6428424f nodeName:}" failed. No retries permitted until 2026-07-07 14:13:58.63358092 +0000 UTC m=+289.631171613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dq65l" (UID: "e9b9bfcb-3ffe-4b06-bdea-a0be6428424f") : configmap "kube-rbac-proxy" not found
Jul 07 14:13:51.021628 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:13:51.021587    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bbxsw" podUID="7bc20856-2dc3-409d-aa1c-92eb8ff59349"
Jul 07 14:13:51.374075 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:51.374046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:13:54.967102 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:54.967063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:13:54.969493 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:54.969474    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bc20856-2dc3-409d-aa1c-92eb8ff59349-metrics-tls\") pod \"dns-default-bbxsw\" (UID: \"7bc20856-2dc3-409d-aa1c-92eb8ff59349\") " pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:13:54.977722 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:54.977693    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2225x\""
Jul 07 14:13:54.985782 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:54.985750    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:13:55.116172 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:55.116143    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bbxsw"]
Jul 07 14:13:55.119440 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:13:55.119415    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc20856_2dc3_409d_aa1c_92eb8ff59349.slice/crio-84dec92d8426f0a143fd3cc9fd3850d04d228f21d610d42ca8ab70704fa605d1 WatchSource:0}: Error finding container 84dec92d8426f0a143fd3cc9fd3850d04d228f21d610d42ca8ab70704fa605d1: Status 404 returned error can't find the container with id 84dec92d8426f0a143fd3cc9fd3850d04d228f21d610d42ca8ab70704fa605d1
Jul 07 14:13:55.386803 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:55.386764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bbxsw" event={"ID":"7bc20856-2dc3-409d-aa1c-92eb8ff59349","Type":"ContainerStarted","Data":"84dec92d8426f0a143fd3cc9fd3850d04d228f21d610d42ca8ab70704fa605d1"}
Jul 07 14:13:57.394980 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:57.394932    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bbxsw" event={"ID":"7bc20856-2dc3-409d-aa1c-92eb8ff59349","Type":"ContainerStarted","Data":"fbd585f2b001d2803b08a34f6c2d3a49318a40f81c3bb225be018bc2ef002690"}
Jul 07 14:13:57.395410 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:57.395013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bbxsw" event={"ID":"7bc20856-2dc3-409d-aa1c-92eb8ff59349","Type":"ContainerStarted","Data":"a646d6f7d890ea7c96b52be607bf89271d370fcd257a14bcdcf06f0507fb8520"}
Jul 07 14:13:57.395410 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:57.395066    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:13:57.415255 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:57.415200    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bbxsw" podStartSLOduration=251.962418973 podStartE2EDuration="4m13.415183978s" totalImagesPullingTime="1.452765005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:13:57.414676262 +0000 UTC m=+288.412266997" watchObservedRunningTime="2026-07-07 14:13:57.415183978 +0000 UTC m=+288.412774692"
Jul 07 14:13:58.696981 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:58.696943    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:58.697367 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:58.697006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:58.697504 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:58.697486    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:58.699313 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:58.699284    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9b9bfcb-3ffe-4b06-bdea-a0be6428424f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dq65l\" (UID: \"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f\") " pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:58.814478 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:58.814444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dq65l"
Jul 07 14:13:58.952000 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:58.951915    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dq65l"]
Jul 07 14:13:58.954754 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:13:58.954727    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b9bfcb_3ffe_4b06_bdea_a0be6428424f.slice/crio-48f1d60e65949c2d4f25f3daa122995b4953666cae0ce63a4f9e2a748d0afc23 WatchSource:0}: Error finding container 48f1d60e65949c2d4f25f3daa122995b4953666cae0ce63a4f9e2a748d0afc23: Status 404 returned error can't find the container with id 48f1d60e65949c2d4f25f3daa122995b4953666cae0ce63a4f9e2a748d0afc23
Jul 07 14:13:59.403273 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:13:59.403231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dq65l" event={"ID":"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f","Type":"ContainerStarted","Data":"48f1d60e65949c2d4f25f3daa122995b4953666cae0ce63a4f9e2a748d0afc23"}
Jul 07 14:14:03.418481 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:03.418446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dq65l" event={"ID":"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f","Type":"ContainerStarted","Data":"2cf3ae1e61db48969076cd3e24b5837963f156bc402e5431726269ce472ded73"}
Jul 07 14:14:03.481551 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:14:03.481509    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-znqh7" podUID="a92bcc45-e37c-46ed-9887-6a895f7e7104"
Jul 07 14:14:04.423957 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:04.423915    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dq65l" event={"ID":"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f","Type":"ContainerStarted","Data":"d89e3388377af5b5ec52d903a2706c1589ebacfb2bdf49cf5bd7560002e385ba"}
Jul 07 14:14:06.432612 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:06.432579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dq65l" event={"ID":"e9b9bfcb-3ffe-4b06-bdea-a0be6428424f","Type":"ContainerStarted","Data":"65ae28d3c3131dae40ad3b40d87304f49cfd56b89398ae7bbe86b8d5cfb6e99f"}
Jul 07 14:14:06.453955 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:06.453899    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dq65l" podStartSLOduration=17.23091503 podStartE2EDuration="24.453879747s" totalImagesPullingTime="7.222964717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:13:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:14:06.453333198 +0000 UTC m=+297.450923913" watchObservedRunningTime="2026-07-07 14:14:06.453879747 +0000 UTC m=+297.451470463"
Jul 07 14:14:07.400426 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:07.400396    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bbxsw"
Jul 07 14:14:09.327749 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.327715    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 14:14:09.716336 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.716301    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-679d4cb8b5-mjvw7"]
Jul 07 14:14:09.719311 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.719295    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.721548 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.721525    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 14:14:09.721700 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.721680    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 14:14:09.721757 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.721722    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6rbz7\""
Jul 07 14:14:09.722470 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.722457    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 14:14:09.731885 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.731859    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 14:14:09.748460 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.748430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-679d4cb8b5-mjvw7"]
Jul 07 14:14:09.775502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.775468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-trusted-ca\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.775668 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.775510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-registry-certificates\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.775668 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.775548    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-bound-sa-token\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.775668 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.775601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-registry-tls\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.775780 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.775667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxmn6\" (UniqueName: \"kubernetes.io/projected/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-kube-api-access-wxmn6\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.775780 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.775716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-installation-pull-secrets\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.775780 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.775758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-image-registry-private-configuration\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.775870 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.775781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-ca-trust-extracted\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.876390 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.876351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-image-registry-private-configuration\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.876390 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.876390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-ca-trust-extracted\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.876657 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.876413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-trusted-ca\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.876657 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.876436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-registry-certificates\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.876657 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.876572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-bound-sa-token\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.876657 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.876645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-registry-tls\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.876860 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.876689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxmn6\" (UniqueName: \"kubernetes.io/projected/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-kube-api-access-wxmn6\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.876860 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.876719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-installation-pull-secrets\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.876965 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.876858    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-ca-trust-extracted\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.877449 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.877423    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-registry-certificates\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.877718 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.877697    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-trusted-ca\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.878963 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.878934    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-installation-pull-secrets\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.879109 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.879092    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-image-registry-private-configuration\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.879170 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.879119    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-registry-tls\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.886188 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.886161    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-bound-sa-token\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:09.886281 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:09.886165    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxmn6\" (UniqueName: \"kubernetes.io/projected/fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927-kube-api-access-wxmn6\") pod \"image-registry-679d4cb8b5-mjvw7\" (UID: \"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927\") " pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:10.027805 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:10.027728    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:10.159752 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:10.159723    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-679d4cb8b5-mjvw7"]
Jul 07 14:14:10.162661 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:14:10.162637    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4b52a9_eef8_4ad9_bd0a_1ba6efc91927.slice/crio-8f2e63d26e20da5b9130d05692926e991e83d7aa395717058fd02692a17999a7 WatchSource:0}: Error finding container 8f2e63d26e20da5b9130d05692926e991e83d7aa395717058fd02692a17999a7: Status 404 returned error can't find the container with id 8f2e63d26e20da5b9130d05692926e991e83d7aa395717058fd02692a17999a7
Jul 07 14:14:10.445532 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:10.445489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7" event={"ID":"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927","Type":"ContainerStarted","Data":"cdeb3e6540f2455aac492c30104e55e91d8b2f30873b3d87b63d9e4668bd7f18"}
Jul 07 14:14:10.445944 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:10.445542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7" event={"ID":"fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927","Type":"ContainerStarted","Data":"8f2e63d26e20da5b9130d05692926e991e83d7aa395717058fd02692a17999a7"}
Jul 07 14:14:10.445944 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:10.445600    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:10.465878 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:10.465817    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7" podStartSLOduration=1.465796073 podStartE2EDuration="1.465796073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:14:10.465361036 +0000 UTC m=+301.462951751" watchObservedRunningTime="2026-07-07 14:14:10.465796073 +0000 UTC m=+301.463386789"
Jul 07 14:14:17.481607 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:17.481570    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:14:21.316680 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.316646    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"]
Jul 07 14:14:21.324286 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.324261    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"]
Jul 07 14:14:21.324458 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.324406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.328048 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.328022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.339169 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.339129    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"]
Jul 07 14:14:21.340147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.340121    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"]
Jul 07 14:14:21.340930 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.340904    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6cphb\""
Jul 07 14:14:21.341076 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.340950    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 14:14:21.341076 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.340950    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 14:14:21.341800 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.341779    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 14:14:21.341965 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.341814    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 14:14:21.341965 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.341863    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 14:14:21.341965 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.341915    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 14:14:21.342269 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.342247    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 14:14:21.342368 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.342347    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xtbff\""
Jul 07 14:14:21.342463 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.342448    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 14:14:21.353875 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.353847    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-77z5d"]
Jul 07 14:14:21.357594 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.357575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.360803 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.360779    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 14:14:21.360940 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.360916    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 14:14:21.361044 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.360984    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z45xp\""
Jul 07 14:14:21.361221 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.361207    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 14:14:21.469976 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.469945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-wtmp\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.469976 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.469979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-tls\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.470284 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.469998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b8fcc8-7f7d-473f-9218-d472e83f1835-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.470284 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69g4\" (UniqueName: \"kubernetes.io/projected/81b8fcc8-7f7d-473f-9218-d472e83f1835-kube-api-access-k69g4\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.470284 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.470284 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470094    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7n4\" (UniqueName: \"kubernetes.io/projected/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-api-access-hn7n4\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.470284 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f9569e9-6251-4009-aabf-9b91b7b459af-metrics-client-ca\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.470284 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81b8fcc8-7f7d-473f-9218-d472e83f1835-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.470284 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.470284 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.470284 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470262    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmq6\" (UniqueName: \"kubernetes.io/projected/4f9569e9-6251-4009-aabf-9b91b7b459af-kube-api-access-dbmq6\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.470686 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f9569e9-6251-4009-aabf-9b91b7b459af-root\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.470686 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.470686 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-textfile\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.470686 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-accelerators-collector-config\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.470686 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f9569e9-6251-4009-aabf-9b91b7b459af-sys\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.470686 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.470686 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81b8fcc8-7f7d-473f-9218-d472e83f1835-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.470686 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.470529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.571975 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.571888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f9569e9-6251-4009-aabf-9b91b7b459af-sys\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.571975 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.571928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.571975 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.571955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81b8fcc8-7f7d-473f-9218-d472e83f1835-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.572229 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.571980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.572229 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572002    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f9569e9-6251-4009-aabf-9b91b7b459af-sys\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.572229 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-wtmp\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.572229 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-tls\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.572229 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b8fcc8-7f7d-473f-9218-d472e83f1835-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.572229 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69g4\" (UniqueName: \"kubernetes.io/projected/81b8fcc8-7f7d-473f-9218-d472e83f1835-kube-api-access-k69g4\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.572229 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.572229 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572160    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-wtmp\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.572649 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7n4\" (UniqueName: \"kubernetes.io/projected/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-api-access-hn7n4\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.572910 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572886    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81b8fcc8-7f7d-473f-9218-d472e83f1835-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.573007 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572884    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.573007 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f9569e9-6251-4009-aabf-9b91b7b459af-metrics-client-ca\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.573007 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.572979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81b8fcc8-7f7d-473f-9218-d472e83f1835-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.573118 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.573021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.573118 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.573052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.573118 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.573095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmq6\" (UniqueName: \"kubernetes.io/projected/4f9569e9-6251-4009-aabf-9b91b7b459af-kube-api-access-dbmq6\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.573491 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.573464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f9569e9-6251-4009-aabf-9b91b7b459af-root\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.573491 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.573481    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f9569e9-6251-4009-aabf-9b91b7b459af-metrics-client-ca\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.573681 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.573509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.573681 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.573552    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f9569e9-6251-4009-aabf-9b91b7b459af-root\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.573681 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.573580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-textfile\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.573681 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.573654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-accelerators-collector-config\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.574376 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.574220    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.574376 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.574234    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-accelerators-collector-config\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.574376 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.574322    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.574578 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.574507    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-textfile\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.575148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.575109    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-tls\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.575148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.575141    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.575314 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.575229    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.575388 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.575369    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81b8fcc8-7f7d-473f-9218-d472e83f1835-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.575898 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.575875    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81b8fcc8-7f7d-473f-9218-d472e83f1835-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.576222 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.576201    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f9569e9-6251-4009-aabf-9b91b7b459af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.587028 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.587000    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69g4\" (UniqueName: \"kubernetes.io/projected/81b8fcc8-7f7d-473f-9218-d472e83f1835-kube-api-access-k69g4\") pod \"openshift-state-metrics-54cf96dfcf-ms8qd\" (UID: \"81b8fcc8-7f7d-473f-9218-d472e83f1835\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.589155 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.589124    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7n4\" (UniqueName: \"kubernetes.io/projected/4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10-kube-api-access-hn7n4\") pod \"kube-state-metrics-7c9dbb8b7d-8hgbl\" (UID: \"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.591314 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.591278    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmq6\" (UniqueName: \"kubernetes.io/projected/4f9569e9-6251-4009-aabf-9b91b7b459af-kube-api-access-dbmq6\") pod \"node-exporter-77z5d\" (UID: \"4f9569e9-6251-4009-aabf-9b91b7b459af\") " pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.635088 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.635045    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"
Jul 07 14:14:21.641254 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.641234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"
Jul 07 14:14:21.666694 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.666661    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77z5d"
Jul 07 14:14:21.681864 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:14:21.681831    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9569e9_6251_4009_aabf_9b91b7b459af.slice/crio-c7cc1a0a82e812fdb4715e5f9e9125c42a252f0a7d00de83981a87813b5dcc5c WatchSource:0}: Error finding container c7cc1a0a82e812fdb4715e5f9e9125c42a252f0a7d00de83981a87813b5dcc5c: Status 404 returned error can't find the container with id c7cc1a0a82e812fdb4715e5f9e9125c42a252f0a7d00de83981a87813b5dcc5c
Jul 07 14:14:21.683696 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.683669    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:14:21.786731 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.786693    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd"]
Jul 07 14:14:21.789681 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:14:21.789649    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b8fcc8_7f7d_473f_9218_d472e83f1835.slice/crio-ef1251b1d748f53f9723f9254bbd3e69173ee59c2a53d231096bdf669855cbd5 WatchSource:0}: Error finding container ef1251b1d748f53f9723f9254bbd3e69173ee59c2a53d231096bdf669855cbd5: Status 404 returned error can't find the container with id ef1251b1d748f53f9723f9254bbd3e69173ee59c2a53d231096bdf669855cbd5
Jul 07 14:14:21.807444 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:21.804719    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl"]
Jul 07 14:14:21.810048 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:14:21.810018    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7cdcd9_aab7_4abd_9a93_ca0f6d76ae10.slice/crio-de3fcf4f7c0787862171a4d9f150f85d114839f299d4d6415cced3b40dc907d6 WatchSource:0}: Error finding container de3fcf4f7c0787862171a4d9f150f85d114839f299d4d6415cced3b40dc907d6: Status 404 returned error can't find the container with id de3fcf4f7c0787862171a4d9f150f85d114839f299d4d6415cced3b40dc907d6
Jul 07 14:14:22.488237 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:22.488195    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd" event={"ID":"81b8fcc8-7f7d-473f-9218-d472e83f1835","Type":"ContainerStarted","Data":"7079a0465e81af8e61c680d6326ba327f9fd422bbd07e8f38a03bd55f1119a4d"}
Jul 07 14:14:22.488743 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:22.488275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd" event={"ID":"81b8fcc8-7f7d-473f-9218-d472e83f1835","Type":"ContainerStarted","Data":"c0dc8b1d89df44aea0da1e168e621233b1d82541bb4339eae04c56285fcd1631"}
Jul 07 14:14:22.488743 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:22.488305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd" event={"ID":"81b8fcc8-7f7d-473f-9218-d472e83f1835","Type":"ContainerStarted","Data":"ef1251b1d748f53f9723f9254bbd3e69173ee59c2a53d231096bdf669855cbd5"}
Jul 07 14:14:22.490233 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:22.490199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77z5d" event={"ID":"4f9569e9-6251-4009-aabf-9b91b7b459af","Type":"ContainerStarted","Data":"c7cc1a0a82e812fdb4715e5f9e9125c42a252f0a7d00de83981a87813b5dcc5c"}
Jul 07 14:14:22.491753 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:22.491719    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl" event={"ID":"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10","Type":"ContainerStarted","Data":"de3fcf4f7c0787862171a4d9f150f85d114839f299d4d6415cced3b40dc907d6"}
Jul 07 14:14:23.496966 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:23.496938    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f9569e9-6251-4009-aabf-9b91b7b459af" containerID="7276c67f80decf2e879c3508e09ead0ba3fa3132362418e5c629d29ac7d2f871" exitCode=0
Jul 07 14:14:23.497404 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:23.496978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77z5d" event={"ID":"4f9569e9-6251-4009-aabf-9b91b7b459af","Type":"ContainerDied","Data":"7276c67f80decf2e879c3508e09ead0ba3fa3132362418e5c629d29ac7d2f871"}
Jul 07 14:14:24.502275 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:24.502241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd" event={"ID":"81b8fcc8-7f7d-473f-9218-d472e83f1835","Type":"ContainerStarted","Data":"9caabb2290e197755aaa7ee27827fd58f5e122c45f4bbaf98b4dfc12b9c83740"}
Jul 07 14:14:24.504301 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:24.504278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77z5d" event={"ID":"4f9569e9-6251-4009-aabf-9b91b7b459af","Type":"ContainerStarted","Data":"7550c47a7c3c21fd96940035adbe6f8adadee3d4d43324c290b64003e6e66b4d"}
Jul 07 14:14:24.504427 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:24.504322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77z5d" event={"ID":"4f9569e9-6251-4009-aabf-9b91b7b459af","Type":"ContainerStarted","Data":"85578f4c3e4b850f76de6514118f320cd0263e14e8b9afdfa3005d4ba5cf3399"}
Jul 07 14:14:24.506322 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:24.506298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl" event={"ID":"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10","Type":"ContainerStarted","Data":"1c2b4adda6ffaf367bbf6729a7b467fbcfce759901184a95e81562a993d28bfd"}
Jul 07 14:14:24.506441 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:24.506350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl" event={"ID":"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10","Type":"ContainerStarted","Data":"d99e4c4efed6c43ec03861cff897a0a4b3f586055c811a7a6192590322082950"}
Jul 07 14:14:24.506441 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:24.506383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl" event={"ID":"4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10","Type":"ContainerStarted","Data":"49d49c515ea4f160d94ec2efce4ed7816b0191b85c03b12615e6a17b022eec18"}
Jul 07 14:14:24.525766 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:24.525711    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-ms8qd" podStartSLOduration=2.010272741 podStartE2EDuration="3.525689222s" totalImagesPullingTime="1.515416481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:14:24.524159075 +0000 UTC m=+315.521749802" watchObservedRunningTime="2026-07-07 14:14:24.525689222 +0000 UTC m=+315.523279957"
Jul 07 14:14:24.546164 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:24.546104    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8hgbl" podStartSLOduration=1.911973212 podStartE2EDuration="3.546087052s" totalImagesPullingTime="1.63411384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:14:24.54474377 +0000 UTC m=+315.542334487" watchObservedRunningTime="2026-07-07 14:14:24.546087052 +0000 UTC m=+315.543677767"
Jul 07 14:14:24.565727 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:24.565668    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-77z5d" podStartSLOduration=2.361591447 podStartE2EDuration="3.565646397s" totalImagesPullingTime="1.129572997s" totalInitContainerRuntime="74.481953ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:14:24.564909714 +0000 UTC m=+315.562500429" watchObservedRunningTime="2026-07-07 14:14:24.565646397 +0000 UTC m=+315.563237108"
Jul 07 14:14:31.453380 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:31.453350    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-679d4cb8b5-mjvw7"
Jul 07 14:14:45.992604 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:45.992560    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" podUID="d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 07 14:14:55.992557 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:14:55.992513    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" podUID="d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 07 14:15:05.992083 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:05.992041    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" podUID="d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 07 14:15:05.992543 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:05.992110    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77"
Jul 07 14:15:05.992587 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:05.992556    2565 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"7e7e79581dfb1aa744eb120b37da7b9505882c6c830d2dc5ea1fc4f6009958e1"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 07 14:15:05.992643 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:05.992606    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" podUID="d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0" containerName="service-proxy" containerID="cri-o://7e7e79581dfb1aa744eb120b37da7b9505882c6c830d2dc5ea1fc4f6009958e1" gracePeriod=30
Jul 07 14:15:06.649736 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:06.649698    2565 generic.go:356] "Generic (PLEG): container finished" podID="d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0" containerID="7e7e79581dfb1aa744eb120b37da7b9505882c6c830d2dc5ea1fc4f6009958e1" exitCode=2
Jul 07 14:15:06.649736 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:06.649731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" event={"ID":"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0","Type":"ContainerDied","Data":"7e7e79581dfb1aa744eb120b37da7b9505882c6c830d2dc5ea1fc4f6009958e1"}
Jul 07 14:15:06.649951 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:06.649777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58b9cd9f69-m6d77" event={"ID":"d6f4b43c-62c2-40a9-b977-7bcd6a17f5b0","Type":"ContainerStarted","Data":"4312b5750cceb0924c571556a2eafa47158e7aae38849be395368fd4105d52d4"}
Jul 07 14:15:08.304364 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:08.304337    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wkpv6_c200e335-efdd-4d8b-bb88-42fa4bfecf13/serve-healthcheck-canary/0.log"
Jul 07 14:15:23.347609 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:23.347552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:15:23.349979 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:23.349954    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92bcc45-e37c-46ed-9887-6a895f7e7104-metrics-certs\") pod \"network-metrics-daemon-znqh7\" (UID: \"a92bcc45-e37c-46ed-9887-6a895f7e7104\") " pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:15:23.486786 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:23.486750    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2hv7\""
Jul 07 14:15:23.492801 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:23.492778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-znqh7"
Jul 07 14:15:23.647849 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:23.647813    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-znqh7"]
Jul 07 14:15:23.651398 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:15:23.651369    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92bcc45_e37c_46ed_9887_6a895f7e7104.slice/crio-2ba5e4f4d40dd3b6dae9497b76f4b8598d0ad984969f024a21e7926af7e61d94 WatchSource:0}: Error finding container 2ba5e4f4d40dd3b6dae9497b76f4b8598d0ad984969f024a21e7926af7e61d94: Status 404 returned error can't find the container with id 2ba5e4f4d40dd3b6dae9497b76f4b8598d0ad984969f024a21e7926af7e61d94
Jul 07 14:15:23.712789 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:23.712755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-znqh7" event={"ID":"a92bcc45-e37c-46ed-9887-6a895f7e7104","Type":"ContainerStarted","Data":"2ba5e4f4d40dd3b6dae9497b76f4b8598d0ad984969f024a21e7926af7e61d94"}
Jul 07 14:15:24.718503 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:24.718461    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-znqh7" event={"ID":"a92bcc45-e37c-46ed-9887-6a895f7e7104","Type":"ContainerStarted","Data":"1c3624f8223b1d07783b48b86da423171f8b71bf7b0dd24783321d7d8f9e65d0"}
Jul 07 14:15:24.718941 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:15:24.718537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-znqh7" event={"ID":"a92bcc45-e37c-46ed-9887-6a895f7e7104","Type":"ContainerStarted","Data":"58507e212b77305312313f5c3fe8364a2c43fed65ec2ec8cd7ddcd227425a3bb"}
Jul 07 14:20:00.263462 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.263388    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-znqh7" podStartSLOduration=650.399464439 podStartE2EDuration="10m51.263350196s" totalImagesPullingTime="863.885757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:15:24.739047835 +0000 UTC m=+375.736638551" watchObservedRunningTime="2026-07-07 14:20:00.263350196 +0000 UTC m=+651.260940905"
Jul 07 14:20:00.264911 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.264885    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"]
Jul 07 14:20:00.267916 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.267898    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:00.270429 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.270402    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 07 14:20:00.270543 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.270440    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 14:20:00.270543 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.270502    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-tfpdh\""
Jul 07 14:20:00.271466 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.271451    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 14:20:00.285480 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.285454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"]
Jul 07 14:20:00.300728 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.300694    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qmqvs"]
Jul 07 14:20:00.303540 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.303524    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:00.306050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.306031    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4s9m8\""
Jul 07 14:20:00.306146 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.306030    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 14:20:00.319311 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.319280    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qmqvs"]
Jul 07 14:20:00.383667 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.383603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8bb4e7b-2dd5-4415-9612-b590440c162d-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xs5z2\" (UID: \"a8bb4e7b-2dd5-4415-9612-b590440c162d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:00.383667 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.383669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5sc2\" (UniqueName: \"kubernetes.io/projected/a8bb4e7b-2dd5-4415-9612-b590440c162d-kube-api-access-d5sc2\") pod \"llmisvc-controller-manager-67d95fdc48-xs5z2\" (UID: \"a8bb4e7b-2dd5-4415-9612-b590440c162d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:00.383880 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.383714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca-data\") pod \"seaweedfs-64568bcd49-qmqvs\" (UID: \"771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca\") " pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:00.383880 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.383754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gd5\" (UniqueName: \"kubernetes.io/projected/771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca-kube-api-access-x5gd5\") pod \"seaweedfs-64568bcd49-qmqvs\" (UID: \"771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca\") " pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:00.484972 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.484886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8bb4e7b-2dd5-4415-9612-b590440c162d-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xs5z2\" (UID: \"a8bb4e7b-2dd5-4415-9612-b590440c162d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:00.485143 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:20:00.485040    2565 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 07 14:20:00.485143 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.485058    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5sc2\" (UniqueName: \"kubernetes.io/projected/a8bb4e7b-2dd5-4415-9612-b590440c162d-kube-api-access-d5sc2\") pod \"llmisvc-controller-manager-67d95fdc48-xs5z2\" (UID: \"a8bb4e7b-2dd5-4415-9612-b590440c162d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:00.485143 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.485083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca-data\") pod \"seaweedfs-64568bcd49-qmqvs\" (UID: \"771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca\") " pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:00.485143 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:20:00.485098    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8bb4e7b-2dd5-4415-9612-b590440c162d-cert podName:a8bb4e7b-2dd5-4415-9612-b590440c162d nodeName:}" failed. No retries permitted until 2026-07-07 14:20:00.985080669 +0000 UTC m=+651.982671362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a8bb4e7b-2dd5-4415-9612-b590440c162d-cert") pod "llmisvc-controller-manager-67d95fdc48-xs5z2" (UID: "a8bb4e7b-2dd5-4415-9612-b590440c162d") : secret "llmisvc-webhook-server-cert" not found
Jul 07 14:20:00.485305 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.485188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gd5\" (UniqueName: \"kubernetes.io/projected/771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca-kube-api-access-x5gd5\") pod \"seaweedfs-64568bcd49-qmqvs\" (UID: \"771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca\") " pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:00.485434 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.485419    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca-data\") pod \"seaweedfs-64568bcd49-qmqvs\" (UID: \"771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca\") " pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:00.501198 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.501162    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gd5\" (UniqueName: \"kubernetes.io/projected/771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca-kube-api-access-x5gd5\") pod \"seaweedfs-64568bcd49-qmqvs\" (UID: \"771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca\") " pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:00.509535 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.509507    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5sc2\" (UniqueName: \"kubernetes.io/projected/a8bb4e7b-2dd5-4415-9612-b590440c162d-kube-api-access-d5sc2\") pod \"llmisvc-controller-manager-67d95fdc48-xs5z2\" (UID: \"a8bb4e7b-2dd5-4415-9612-b590440c162d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:00.612854 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.612759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:00.747821 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.747788    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qmqvs"]
Jul 07 14:20:00.751257 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:20:00.751231    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771a59f7_d5c9_4b4d_a2f4_e04e6bd251ca.slice/crio-b2176ae5b3218721f19ff52ac15f065be910230ea235aaaa4c1ea05350c49b1d WatchSource:0}: Error finding container b2176ae5b3218721f19ff52ac15f065be910230ea235aaaa4c1ea05350c49b1d: Status 404 returned error can't find the container with id b2176ae5b3218721f19ff52ac15f065be910230ea235aaaa4c1ea05350c49b1d
Jul 07 14:20:00.752526 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.752512    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 14:20:00.990345 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.990301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8bb4e7b-2dd5-4415-9612-b590440c162d-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xs5z2\" (UID: \"a8bb4e7b-2dd5-4415-9612-b590440c162d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:00.992653 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:00.992637    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8bb4e7b-2dd5-4415-9612-b590440c162d-cert\") pod \"llmisvc-controller-manager-67d95fdc48-xs5z2\" (UID: \"a8bb4e7b-2dd5-4415-9612-b590440c162d\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:01.178115 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:01.178072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:01.315607 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:01.315575    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"]
Jul 07 14:20:01.319151 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:20:01.319119    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda8bb4e7b_2dd5_4415_9612_b590440c162d.slice/crio-a19cbea5e2c07454840056b6112196177ac7713dee003383e8f24ebb6b58c54c WatchSource:0}: Error finding container a19cbea5e2c07454840056b6112196177ac7713dee003383e8f24ebb6b58c54c: Status 404 returned error can't find the container with id a19cbea5e2c07454840056b6112196177ac7713dee003383e8f24ebb6b58c54c
Jul 07 14:20:01.662922 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:01.662887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qmqvs" event={"ID":"771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca","Type":"ContainerStarted","Data":"b2176ae5b3218721f19ff52ac15f065be910230ea235aaaa4c1ea05350c49b1d"}
Jul 07 14:20:01.664162 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:01.664132    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2" event={"ID":"a8bb4e7b-2dd5-4415-9612-b590440c162d","Type":"ContainerStarted","Data":"a19cbea5e2c07454840056b6112196177ac7713dee003383e8f24ebb6b58c54c"}
Jul 07 14:20:06.685847 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:06.685805    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qmqvs" event={"ID":"771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca","Type":"ContainerStarted","Data":"d6a17cd29cd6236647c337ceb3a28e2369a09e57c2a83dd5f4e2c25076907595"}
Jul 07 14:20:06.686455 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:06.685873    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:06.687297 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:06.687275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2" event={"ID":"a8bb4e7b-2dd5-4415-9612-b590440c162d","Type":"ContainerStarted","Data":"f1826d08186223cd6313d6c18a89dc58827341d1e0a119958103c3db324d312b"}
Jul 07 14:20:06.687399 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:06.687389    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:20:06.706126 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:06.706068    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qmqvs" podStartSLOduration=1.3120773350000001 podStartE2EDuration="6.706048874s" totalImagesPullingTime="5.393971539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:20:06.705672682 +0000 UTC m=+657.703263396" watchObservedRunningTime="2026-07-07 14:20:06.706048874 +0000 UTC m=+657.703639590"
Jul 07 14:20:06.724178 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:06.724118    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2" podStartSLOduration=1.9455843929999999 podStartE2EDuration="6.72410012s" totalImagesPullingTime="4.778515727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:20:06.723312105 +0000 UTC m=+657.720902820" watchObservedRunningTime="2026-07-07 14:20:06.72410012 +0000 UTC m=+657.721690834"
Jul 07 14:20:12.693605 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:12.693573    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qmqvs"
Jul 07 14:20:37.693548 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:20:37.693518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-xs5z2"
Jul 07 14:21:13.616893 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.616853    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-jzz99"]
Jul 07 14:21:13.619984 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.619962    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:21:13.631078 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.631055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06bcbd16-2cac-4ecb-b017-12d1f26f7b14-cert\") pod \"odh-model-controller-84b54b7f97-jzz99\" (UID: \"06bcbd16-2cac-4ecb-b017-12d1f26f7b14\") " pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:21:13.631186 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.631094    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2v6n\" (UniqueName: \"kubernetes.io/projected/06bcbd16-2cac-4ecb-b017-12d1f26f7b14-kube-api-access-s2v6n\") pod \"odh-model-controller-84b54b7f97-jzz99\" (UID: \"06bcbd16-2cac-4ecb-b017-12d1f26f7b14\") " pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:21:13.658110 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.658079    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-pn2m4\""
Jul 07 14:21:13.658249 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.658117    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 07 14:21:13.671019 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.670973    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-jzz99"]
Jul 07 14:21:13.732199 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.732162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06bcbd16-2cac-4ecb-b017-12d1f26f7b14-cert\") pod \"odh-model-controller-84b54b7f97-jzz99\" (UID: \"06bcbd16-2cac-4ecb-b017-12d1f26f7b14\") " pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:21:13.732367 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.732208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2v6n\" (UniqueName: \"kubernetes.io/projected/06bcbd16-2cac-4ecb-b017-12d1f26f7b14-kube-api-access-s2v6n\") pod \"odh-model-controller-84b54b7f97-jzz99\" (UID: \"06bcbd16-2cac-4ecb-b017-12d1f26f7b14\") " pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:21:13.734717 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.734680    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06bcbd16-2cac-4ecb-b017-12d1f26f7b14-cert\") pod \"odh-model-controller-84b54b7f97-jzz99\" (UID: \"06bcbd16-2cac-4ecb-b017-12d1f26f7b14\") " pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:21:13.748574 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.748542    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2v6n\" (UniqueName: \"kubernetes.io/projected/06bcbd16-2cac-4ecb-b017-12d1f26f7b14-kube-api-access-s2v6n\") pod \"odh-model-controller-84b54b7f97-jzz99\" (UID: \"06bcbd16-2cac-4ecb-b017-12d1f26f7b14\") " pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:21:13.929775 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:13.929741    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:21:14.077226 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:14.077198    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-jzz99"]
Jul 07 14:21:14.079598 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:21:14.079568    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bcbd16_2cac_4ecb_b017_12d1f26f7b14.slice/crio-1ad33a088e46684af742553e9aa69ac8a8b281b592d0922cc30e1bc23ecab6d7 WatchSource:0}: Error finding container 1ad33a088e46684af742553e9aa69ac8a8b281b592d0922cc30e1bc23ecab6d7: Status 404 returned error can't find the container with id 1ad33a088e46684af742553e9aa69ac8a8b281b592d0922cc30e1bc23ecab6d7
Jul 07 14:21:14.929431 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:14.929380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-jzz99" event={"ID":"06bcbd16-2cac-4ecb-b017-12d1f26f7b14","Type":"ContainerStarted","Data":"1ad33a088e46684af742553e9aa69ac8a8b281b592d0922cc30e1bc23ecab6d7"}
Jul 07 14:21:16.939047 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:16.939013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-jzz99" event={"ID":"06bcbd16-2cac-4ecb-b017-12d1f26f7b14","Type":"ContainerStarted","Data":"9789cf8b00ddf2751cc3c2524f3fa12a8eb515f6d1581ed7b8eeef9531e9df99"}
Jul 07 14:21:16.939426 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:16.939150    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:21:16.970295 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:16.970239    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-jzz99" podStartSLOduration=1.52252718 podStartE2EDuration="3.970220217s" totalImagesPullingTime="2.447693037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:21:16.967293752 +0000 UTC m=+727.964884486" watchObservedRunningTime="2026-07-07 14:21:16.970220217 +0000 UTC m=+727.967810932"
Jul 07 14:21:27.945777 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:21:27.945742    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-jzz99"
Jul 07 14:23:00.617313 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.617254    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"]
Jul 07 14:23:00.620408 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.620390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:00.623489 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.623464    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a2dea-predictor-serving-cert\""
Jul 07 14:23:00.624441 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.624423    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 14:23:00.624441 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.624438    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a2dea-kube-rbac-proxy-sar-config\""
Jul 07 14:23:00.624592 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.624429    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j545g\""
Jul 07 14:23:00.624592 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.624426    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 14:23:00.633806 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.633782    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"]
Jul 07 14:23:00.658204 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.658170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnrr9\" (UniqueName: \"kubernetes.io/projected/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-api-access-nnrr9\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:00.658204 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.658209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-proxy-tls\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:00.658426 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.658232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:00.759744 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.759698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrr9\" (UniqueName: \"kubernetes.io/projected/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-api-access-nnrr9\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:00.759957 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.759754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-proxy-tls\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:00.759957 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.759786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:00.759957 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:23:00.759898    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-serving-cert: secret "success-200-isvc-a2dea-predictor-serving-cert" not found
Jul 07 14:23:00.760124 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:23:00.759987    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-proxy-tls podName:567dd16a-6cf4-4d8c-92e4-9f1ceec627ef nodeName:}" failed. No retries permitted until 2026-07-07 14:23:01.259971873 +0000 UTC m=+832.257562570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-proxy-tls") pod "success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" (UID: "567dd16a-6cf4-4d8c-92e4-9f1ceec627ef") : secret "success-200-isvc-a2dea-predictor-serving-cert" not found
Jul 07 14:23:00.760507 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.760481    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:00.769265 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:00.769237    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrr9\" (UniqueName: \"kubernetes.io/projected/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-api-access-nnrr9\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:01.264978 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:01.264943    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-proxy-tls\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:01.267424 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:01.267399    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-proxy-tls\") pod \"success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") " pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:01.531021 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:01.530916    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:01.691644 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:01.691581    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"]
Jul 07 14:23:01.696881 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:23:01.696849    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567dd16a_6cf4_4d8c_92e4_9f1ceec627ef.slice/crio-508aac549521465d947596dbbfb0cee7583a43999a42a2b2cdd7486a12814ae3 WatchSource:0}: Error finding container 508aac549521465d947596dbbfb0cee7583a43999a42a2b2cdd7486a12814ae3: Status 404 returned error can't find the container with id 508aac549521465d947596dbbfb0cee7583a43999a42a2b2cdd7486a12814ae3
Jul 07 14:23:02.316587 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:02.316546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" event={"ID":"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef","Type":"ContainerStarted","Data":"508aac549521465d947596dbbfb0cee7583a43999a42a2b2cdd7486a12814ae3"}
Jul 07 14:23:15.382114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:15.382079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" event={"ID":"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef","Type":"ContainerStarted","Data":"ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a"}
Jul 07 14:23:18.396794 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:18.396752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" event={"ID":"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef","Type":"ContainerStarted","Data":"4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb"}
Jul 07 14:23:18.397272 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:18.396892    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:18.418845 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:18.418790    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" podStartSLOduration=2.675066765 podStartE2EDuration="18.418772855s" totalImagesPullingTime="15.74370609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:23:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:23:18.417730316 +0000 UTC m=+849.415321031" watchObservedRunningTime="2026-07-07 14:23:18.418772855 +0000 UTC m=+849.416363569"
Jul 07 14:23:19.400220 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:19.400186    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:19.401428 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:19.401403    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 14:23:20.404248 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:20.404209    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 14:23:25.410537 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:25.410509    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:23:25.411048 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:25.411020    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 14:23:35.411250 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:35.411207    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 14:23:45.411487 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:45.411448    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 14:23:55.411437 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:23:55.411396    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 07 14:24:05.411847 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:05.411808    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:24:20.642196 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.642156    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"]
Jul 07 14:24:20.645248 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.645227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:20.648458 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.648433    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a2dea-serving-cert\""
Jul 07 14:24:20.648682 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.648665    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a2dea-kube-rbac-proxy-sar-config\""
Jul 07 14:24:20.655806 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.655781    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"]
Jul 07 14:24:20.707801 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.707763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66870c1d-06f9-42b0-976f-eae6b1256da3-proxy-tls\") pod \"switch-graph-a2dea-694795ff54-nc86f\" (UID: \"66870c1d-06f9-42b0-976f-eae6b1256da3\") " pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:20.707979 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.707817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66870c1d-06f9-42b0-976f-eae6b1256da3-openshift-service-ca-bundle\") pod \"switch-graph-a2dea-694795ff54-nc86f\" (UID: \"66870c1d-06f9-42b0-976f-eae6b1256da3\") " pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:20.808349 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.808303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66870c1d-06f9-42b0-976f-eae6b1256da3-openshift-service-ca-bundle\") pod \"switch-graph-a2dea-694795ff54-nc86f\" (UID: \"66870c1d-06f9-42b0-976f-eae6b1256da3\") " pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:20.808570 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.808391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66870c1d-06f9-42b0-976f-eae6b1256da3-proxy-tls\") pod \"switch-graph-a2dea-694795ff54-nc86f\" (UID: \"66870c1d-06f9-42b0-976f-eae6b1256da3\") " pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:20.809167 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.809143    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66870c1d-06f9-42b0-976f-eae6b1256da3-openshift-service-ca-bundle\") pod \"switch-graph-a2dea-694795ff54-nc86f\" (UID: \"66870c1d-06f9-42b0-976f-eae6b1256da3\") " pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:20.810858 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.810839    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66870c1d-06f9-42b0-976f-eae6b1256da3-proxy-tls\") pod \"switch-graph-a2dea-694795ff54-nc86f\" (UID: \"66870c1d-06f9-42b0-976f-eae6b1256da3\") " pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:20.955801 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:20.955707    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:21.085824 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:21.085791    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"]
Jul 07 14:24:21.088944 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:24:21.088914    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66870c1d_06f9_42b0_976f_eae6b1256da3.slice/crio-f091224980f7426f064c65a4a1d62c6b13d07e5a4ff0f1a4559b0683bb491c64 WatchSource:0}: Error finding container f091224980f7426f064c65a4a1d62c6b13d07e5a4ff0f1a4559b0683bb491c64: Status 404 returned error can't find the container with id f091224980f7426f064c65a4a1d62c6b13d07e5a4ff0f1a4559b0683bb491c64
Jul 07 14:24:21.627679 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:21.627643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" event={"ID":"66870c1d-06f9-42b0-976f-eae6b1256da3","Type":"ContainerStarted","Data":"f091224980f7426f064c65a4a1d62c6b13d07e5a4ff0f1a4559b0683bb491c64"}
Jul 07 14:24:23.637388 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:23.637349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" event={"ID":"66870c1d-06f9-42b0-976f-eae6b1256da3","Type":"ContainerStarted","Data":"f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f"}
Jul 07 14:24:23.637790 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:23.637419    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:23.659541 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:23.659490    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" podStartSLOduration=1.7290720849999999 podStartE2EDuration="3.659473434s" totalImagesPullingTime="1.930401349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:23.658696899 +0000 UTC m=+914.656287615" watchObservedRunningTime="2026-07-07 14:24:23.659473434 +0000 UTC m=+914.657064149"
Jul 07 14:24:29.648382 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:29.648353    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:30.826886 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:30.826855    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"]
Jul 07 14:24:30.827271 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:30.827025    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea" containerID="cri-o://f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f" gracePeriod=30
Jul 07 14:24:30.981234 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:30.981197    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"]
Jul 07 14:24:30.981493 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:30.981460    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container" containerID="cri-o://ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a" gracePeriod=30
Jul 07 14:24:30.981592 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:30.981511    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kube-rbac-proxy" containerID="cri-o://4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb" gracePeriod=30
Jul 07 14:24:31.004506 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.004473    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"]
Jul 07 14:24:31.008047 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.008009    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.010704 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.010676    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-520df-predictor-serving-cert\""
Jul 07 14:24:31.010825 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.010785    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-520df-kube-rbac-proxy-sar-config\""
Jul 07 14:24:31.022568 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.022541    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"]
Jul 07 14:24:31.091177 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.091103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj62n\" (UniqueName: \"kubernetes.io/projected/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-api-access-lj62n\") pod \"success-200-isvc-520df-predictor-86849fb99-w5khm\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") " pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.091177 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.091160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12825a42-a286-48f8-b8cf-493a38b1b7b0-proxy-tls\") pod \"success-200-isvc-520df-predictor-86849fb99-w5khm\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") " pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.091349 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.091245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-520df-predictor-86849fb99-w5khm\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") " pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.191830 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.191784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-520df-predictor-86849fb99-w5khm\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") " pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.192039 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.191856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj62n\" (UniqueName: \"kubernetes.io/projected/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-api-access-lj62n\") pod \"success-200-isvc-520df-predictor-86849fb99-w5khm\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") " pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.192039 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.191922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12825a42-a286-48f8-b8cf-493a38b1b7b0-proxy-tls\") pod \"success-200-isvc-520df-predictor-86849fb99-w5khm\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") " pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.192557 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.192533    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-520df-predictor-86849fb99-w5khm\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") " pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.194349 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.194325    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12825a42-a286-48f8-b8cf-493a38b1b7b0-proxy-tls\") pod \"success-200-isvc-520df-predictor-86849fb99-w5khm\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") " pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.208717 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.208685    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj62n\" (UniqueName: \"kubernetes.io/projected/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-api-access-lj62n\") pod \"success-200-isvc-520df-predictor-86849fb99-w5khm\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") " pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.318884 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.318843    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.452850 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.452801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"]
Jul 07 14:24:31.456228 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:24:31.456200    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12825a42_a286_48f8_b8cf_493a38b1b7b0.slice/crio-254ba14939212566b076295a23b9efbbf49f019054175ed038ff4bbac9c2ed37 WatchSource:0}: Error finding container 254ba14939212566b076295a23b9efbbf49f019054175ed038ff4bbac9c2ed37: Status 404 returned error can't find the container with id 254ba14939212566b076295a23b9efbbf49f019054175ed038ff4bbac9c2ed37
Jul 07 14:24:31.669422 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.669388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" event={"ID":"12825a42-a286-48f8-b8cf-493a38b1b7b0","Type":"ContainerStarted","Data":"48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679"}
Jul 07 14:24:31.669600 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.669462    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.669600 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.669489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" event={"ID":"12825a42-a286-48f8-b8cf-493a38b1b7b0","Type":"ContainerStarted","Data":"7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1"}
Jul 07 14:24:31.669600 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.669517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" event={"ID":"12825a42-a286-48f8-b8cf-493a38b1b7b0","Type":"ContainerStarted","Data":"254ba14939212566b076295a23b9efbbf49f019054175ed038ff4bbac9c2ed37"}
Jul 07 14:24:31.669600 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.669548    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:31.671256 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.671226    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 14:24:31.671355 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.671311    2565 generic.go:356] "Generic (PLEG): container finished" podID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerID="4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb" exitCode=2
Jul 07 14:24:31.671401 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.671379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" event={"ID":"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef","Type":"ContainerDied","Data":"4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb"}
Jul 07 14:24:31.690567 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:31.690514    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podStartSLOduration=1.690496285 podStartE2EDuration="1.690496285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:31.689189055 +0000 UTC m=+922.686779781" watchObservedRunningTime="2026-07-07 14:24:31.690496285 +0000 UTC m=+922.688087000"
Jul 07 14:24:32.675935 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:32.675890    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 14:24:34.338509 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.338479    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:24:34.418258 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.418205    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-rbac-proxy-sar-config\") pod \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") "
Jul 07 14:24:34.418451 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.418311    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-proxy-tls\") pod \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") "
Jul 07 14:24:34.418451 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.418361    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnrr9\" (UniqueName: \"kubernetes.io/projected/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-api-access-nnrr9\") pod \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\" (UID: \"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef\") "
Jul 07 14:24:34.418648 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.418596    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" (UID: "567dd16a-6cf4-4d8c-92e4-9f1ceec627ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:24:34.420551 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.420521    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" (UID: "567dd16a-6cf4-4d8c-92e4-9f1ceec627ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:24:34.420681 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.420563    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-api-access-nnrr9" (OuterVolumeSpecName: "kube-api-access-nnrr9") pod "567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" (UID: "567dd16a-6cf4-4d8c-92e4-9f1ceec627ef"). InnerVolumeSpecName "kube-api-access-nnrr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:24:34.519025 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.518913    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnrr9\" (UniqueName: \"kubernetes.io/projected/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-api-access-nnrr9\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:24:34.519025 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.518959    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:24:34.519025 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.518969    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:24:34.648388 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.648340    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:34.685930 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.685892    2565 generic.go:356] "Generic (PLEG): container finished" podID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerID="ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a" exitCode=0
Jul 07 14:24:34.686104 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.685939    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"
Jul 07 14:24:34.686104 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.685973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" event={"ID":"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef","Type":"ContainerDied","Data":"ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a"}
Jul 07 14:24:34.686104 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.686036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt" event={"ID":"567dd16a-6cf4-4d8c-92e4-9f1ceec627ef","Type":"ContainerDied","Data":"508aac549521465d947596dbbfb0cee7583a43999a42a2b2cdd7486a12814ae3"}
Jul 07 14:24:34.686104 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.686063    2565 scope.go:117] "RemoveContainer" containerID="4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb"
Jul 07 14:24:34.695180 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.695158    2565 scope.go:117] "RemoveContainer" containerID="ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a"
Jul 07 14:24:34.703073 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.703050    2565 scope.go:117] "RemoveContainer" containerID="4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb"
Jul 07 14:24:34.703404 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:24:34.703380    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb\": container with ID starting with 4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb not found: ID does not exist" containerID="4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb"
Jul 07 14:24:34.703470 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.703414    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb"} err="failed to get container status \"4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb\": rpc error: code = NotFound desc = could not find container \"4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb\": container with ID starting with 4d81f45cb95b70e093ede2194730c95481a96cfea6f191885807590346ef1deb not found: ID does not exist"
Jul 07 14:24:34.703470 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.703434    2565 scope.go:117] "RemoveContainer" containerID="ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a"
Jul 07 14:24:34.703798 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:24:34.703780    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a\": container with ID starting with ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a not found: ID does not exist" containerID="ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a"
Jul 07 14:24:34.703865 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.703808    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a"} err="failed to get container status \"ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a\": rpc error: code = NotFound desc = could not find container \"ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a\": container with ID starting with ae7fc2a00ad036cba33aa83557d1f747081b7e3a4e0a44ed00c458008aeb4d1a not found: ID does not exist"
Jul 07 14:24:34.707536 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.707507    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"]
Jul 07 14:24:34.709740 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:34.709715    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a2dea-predictor-68b6fb989f-wb6nt"]
Jul 07 14:24:35.485767 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:35.485736    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" path="/var/lib/kubelet/pods/567dd16a-6cf4-4d8c-92e4-9f1ceec627ef/volumes"
Jul 07 14:24:37.681652 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:37.681607    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:24:37.682147 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:37.682123    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 14:24:39.646200 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:39.646151    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:44.646332 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:44.646280    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:44.646806 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:44.646421    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:24:47.682323 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:47.682273    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 14:24:49.646122 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:49.646087    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:54.646285 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:54.646244    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:24:57.682838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:57.682792    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 14:24:59.646258 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:24:59.646212    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:00.973157 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:00.973130    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:25:01.032900 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.032867    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66870c1d-06f9-42b0-976f-eae6b1256da3-proxy-tls\") pod \"66870c1d-06f9-42b0-976f-eae6b1256da3\" (UID: \"66870c1d-06f9-42b0-976f-eae6b1256da3\") "
Jul 07 14:25:01.033077 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.032952    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66870c1d-06f9-42b0-976f-eae6b1256da3-openshift-service-ca-bundle\") pod \"66870c1d-06f9-42b0-976f-eae6b1256da3\" (UID: \"66870c1d-06f9-42b0-976f-eae6b1256da3\") "
Jul 07 14:25:01.033327 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.033304    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66870c1d-06f9-42b0-976f-eae6b1256da3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "66870c1d-06f9-42b0-976f-eae6b1256da3" (UID: "66870c1d-06f9-42b0-976f-eae6b1256da3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:25:01.034999 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.034972    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66870c1d-06f9-42b0-976f-eae6b1256da3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66870c1d-06f9-42b0-976f-eae6b1256da3" (UID: "66870c1d-06f9-42b0-976f-eae6b1256da3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:25:01.134016 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.133981    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66870c1d-06f9-42b0-976f-eae6b1256da3-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:25:01.134016 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.134014    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66870c1d-06f9-42b0-976f-eae6b1256da3-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:25:01.788597 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.788502    2565 generic.go:356] "Generic (PLEG): container finished" podID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerID="f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f" exitCode=0
Jul 07 14:25:01.788597 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.788541    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" event={"ID":"66870c1d-06f9-42b0-976f-eae6b1256da3","Type":"ContainerDied","Data":"f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f"}
Jul 07 14:25:01.788597 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.788556    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"
Jul 07 14:25:01.788597 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.788584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f" event={"ID":"66870c1d-06f9-42b0-976f-eae6b1256da3","Type":"ContainerDied","Data":"f091224980f7426f064c65a4a1d62c6b13d07e5a4ff0f1a4559b0683bb491c64"}
Jul 07 14:25:01.788960 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.788605    2565 scope.go:117] "RemoveContainer" containerID="f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f"
Jul 07 14:25:01.796468 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.796447    2565 scope.go:117] "RemoveContainer" containerID="f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f"
Jul 07 14:25:01.796772 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:25:01.796752    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f\": container with ID starting with f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f not found: ID does not exist" containerID="f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f"
Jul 07 14:25:01.796831 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.796783    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f"} err="failed to get container status \"f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f\": rpc error: code = NotFound desc = could not find container \"f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f\": container with ID starting with f52a6a706f8e209f1468454b540e41acdcfbb973066bd047f7f327a9a4bd2d6f not found: ID does not exist"
Jul 07 14:25:01.806977 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.806948    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"]
Jul 07 14:25:01.811196 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:01.811167    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a2dea-694795ff54-nc86f"]
Jul 07 14:25:03.486637 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:03.486584    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" path="/var/lib/kubelet/pods/66870c1d-06f9-42b0-976f-eae6b1256da3/volumes"
Jul 07 14:25:07.682098 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:07.682059    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 14:25:17.682213 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:17.682168    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 14:25:20.729788 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.729750    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"]
Jul 07 14:25:20.730194 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.730030    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea"
Jul 07 14:25:20.730194 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.730041    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea"
Jul 07 14:25:20.730194 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.730051    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kube-rbac-proxy"
Jul 07 14:25:20.730194 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.730056    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kube-rbac-proxy"
Jul 07 14:25:20.730194 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.730095    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container"
Jul 07 14:25:20.730194 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.730101    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container"
Jul 07 14:25:20.730194 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.730150    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kube-rbac-proxy"
Jul 07 14:25:20.730194 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.730157    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="567dd16a-6cf4-4d8c-92e4-9f1ceec627ef" containerName="kserve-container"
Jul 07 14:25:20.730194 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.730164    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66870c1d-06f9-42b0-976f-eae6b1256da3" containerName="switch-graph-a2dea"
Jul 07 14:25:20.734285 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.734261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:20.738534 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.738510    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 07 14:25:20.738679 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.738517    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 07 14:25:20.745490 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.745463    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"]
Jul 07 14:25:20.768743 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.768706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd44486-5832-447e-a7da-ed2a1be0f4bb-openshift-service-ca-bundle\") pod \"model-chainer-fcb978798-2vdlf\" (UID: \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\") " pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:20.768922 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.768765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fd44486-5832-447e-a7da-ed2a1be0f4bb-proxy-tls\") pod \"model-chainer-fcb978798-2vdlf\" (UID: \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\") " pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:20.870211 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.870178    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd44486-5832-447e-a7da-ed2a1be0f4bb-openshift-service-ca-bundle\") pod \"model-chainer-fcb978798-2vdlf\" (UID: \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\") " pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:20.870386 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.870223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fd44486-5832-447e-a7da-ed2a1be0f4bb-proxy-tls\") pod \"model-chainer-fcb978798-2vdlf\" (UID: \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\") " pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:20.870386 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:25:20.870332    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 07 14:25:20.870460 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:25:20.870417    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd44486-5832-447e-a7da-ed2a1be0f4bb-proxy-tls podName:7fd44486-5832-447e-a7da-ed2a1be0f4bb nodeName:}" failed. No retries permitted until 2026-07-07 14:25:21.370395361 +0000 UTC m=+972.367986059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7fd44486-5832-447e-a7da-ed2a1be0f4bb-proxy-tls") pod "model-chainer-fcb978798-2vdlf" (UID: "7fd44486-5832-447e-a7da-ed2a1be0f4bb") : secret "model-chainer-serving-cert" not found
Jul 07 14:25:20.871003 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:20.870984    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd44486-5832-447e-a7da-ed2a1be0f4bb-openshift-service-ca-bundle\") pod \"model-chainer-fcb978798-2vdlf\" (UID: \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\") " pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:21.373212 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:21.373178    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fd44486-5832-447e-a7da-ed2a1be0f4bb-proxy-tls\") pod \"model-chainer-fcb978798-2vdlf\" (UID: \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\") " pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:21.375575 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:21.375554    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fd44486-5832-447e-a7da-ed2a1be0f4bb-proxy-tls\") pod \"model-chainer-fcb978798-2vdlf\" (UID: \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\") " pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:21.644664 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:21.644540    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:21.800850 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:21.800821    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"]
Jul 07 14:25:21.803456 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:25:21.803428    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd44486_5832_447e_a7da_ed2a1be0f4bb.slice/crio-38f028abc7aabe7ab06b04cad4b81ec87df15ed18642db27092aa2800cfab3f2 WatchSource:0}: Error finding container 38f028abc7aabe7ab06b04cad4b81ec87df15ed18642db27092aa2800cfab3f2: Status 404 returned error can't find the container with id 38f028abc7aabe7ab06b04cad4b81ec87df15ed18642db27092aa2800cfab3f2
Jul 07 14:25:21.870586 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:21.870542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" event={"ID":"7fd44486-5832-447e-a7da-ed2a1be0f4bb","Type":"ContainerStarted","Data":"f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23"}
Jul 07 14:25:21.870778 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:21.870608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" event={"ID":"7fd44486-5832-447e-a7da-ed2a1be0f4bb","Type":"ContainerStarted","Data":"38f028abc7aabe7ab06b04cad4b81ec87df15ed18642db27092aa2800cfab3f2"}
Jul 07 14:25:21.870778 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:21.870766    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:21.896570 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:21.896468    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" podStartSLOduration=1.896450158 podStartE2EDuration="1.896450158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:21.895851503 +0000 UTC m=+972.893442218" watchObservedRunningTime="2026-07-07 14:25:21.896450158 +0000 UTC m=+972.894040872"
Jul 07 14:25:27.682806 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:27.682767    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:25:27.880672 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:27.880612    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:30.803879 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:30.803841    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"]
Jul 07 14:25:30.804683 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:30.804096    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer" containerID="cri-o://f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23" gracePeriod=30
Jul 07 14:25:31.116806 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.116685    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"]
Jul 07 14:25:31.120129 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.120102    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.122808 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.122786    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ab362-predictor-serving-cert\""
Jul 07 14:25:31.122917 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.122789    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ab362-kube-rbac-proxy-sar-config\""
Jul 07 14:25:31.136849 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.136821    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"]
Jul 07 14:25:31.150462 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.150428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83654928-fe75-4138-90ed-3acf87dad1ae-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.150652 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.150499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrjq\" (UniqueName: \"kubernetes.io/projected/83654928-fe75-4138-90ed-3acf87dad1ae-kube-api-access-qqrjq\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.150652 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.150528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83654928-fe75-4138-90ed-3acf87dad1ae-proxy-tls\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.251949 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.251917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrjq\" (UniqueName: \"kubernetes.io/projected/83654928-fe75-4138-90ed-3acf87dad1ae-kube-api-access-qqrjq\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.251949 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.251953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83654928-fe75-4138-90ed-3acf87dad1ae-proxy-tls\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.252151 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.252000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83654928-fe75-4138-90ed-3acf87dad1ae-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.252151 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:25:31.252115    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ab362-predictor-serving-cert: secret "success-200-isvc-ab362-predictor-serving-cert" not found
Jul 07 14:25:31.252217 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:25:31.252186    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83654928-fe75-4138-90ed-3acf87dad1ae-proxy-tls podName:83654928-fe75-4138-90ed-3acf87dad1ae nodeName:}" failed. No retries permitted until 2026-07-07 14:25:31.752165358 +0000 UTC m=+982.749756054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/83654928-fe75-4138-90ed-3acf87dad1ae-proxy-tls") pod "success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" (UID: "83654928-fe75-4138-90ed-3acf87dad1ae") : secret "success-200-isvc-ab362-predictor-serving-cert" not found
Jul 07 14:25:31.252755 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.252664    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83654928-fe75-4138-90ed-3acf87dad1ae-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.263582 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.263557    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrjq\" (UniqueName: \"kubernetes.io/projected/83654928-fe75-4138-90ed-3acf87dad1ae-kube-api-access-qqrjq\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.756108 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.756065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83654928-fe75-4138-90ed-3acf87dad1ae-proxy-tls\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:31.758455 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:31.758425    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83654928-fe75-4138-90ed-3acf87dad1ae-proxy-tls\") pod \"success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") " pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:32.030326 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:32.030236    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:32.170958 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:32.170929    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"]
Jul 07 14:25:32.174429 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:25:32.174400    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83654928_fe75_4138_90ed_3acf87dad1ae.slice/crio-53a98d0e292cbe1adf5c7490791a8185a193ececc17559f97d86908792c9ac3e WatchSource:0}: Error finding container 53a98d0e292cbe1adf5c7490791a8185a193ececc17559f97d86908792c9ac3e: Status 404 returned error can't find the container with id 53a98d0e292cbe1adf5c7490791a8185a193ececc17559f97d86908792c9ac3e
Jul 07 14:25:32.878339 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:32.878295    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:32.911574 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:32.911537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" event={"ID":"83654928-fe75-4138-90ed-3acf87dad1ae","Type":"ContainerStarted","Data":"47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4"}
Jul 07 14:25:32.911770 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:32.911597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" event={"ID":"83654928-fe75-4138-90ed-3acf87dad1ae","Type":"ContainerStarted","Data":"18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1"}
Jul 07 14:25:32.911770 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:32.911643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" event={"ID":"83654928-fe75-4138-90ed-3acf87dad1ae","Type":"ContainerStarted","Data":"53a98d0e292cbe1adf5c7490791a8185a193ececc17559f97d86908792c9ac3e"}
Jul 07 14:25:32.911770 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:32.911758    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:32.936934 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:32.936882    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" podStartSLOduration=1.93686666 podStartE2EDuration="1.93686666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:32.935139435 +0000 UTC m=+983.932730171" watchObservedRunningTime="2026-07-07 14:25:32.93686666 +0000 UTC m=+983.934457375"
Jul 07 14:25:33.915264 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:33.915229    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:33.916578 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:33.916549    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 14:25:34.918545 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:34.918501    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 14:25:37.878255 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:37.878211    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:39.923411 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:39.923376    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:25:39.924030 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:39.923998    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 14:25:41.102553 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.102519    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"]
Jul 07 14:25:41.105787 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.105763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:41.109838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.109818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-520df-kube-rbac-proxy-sar-config\""
Jul 07 14:25:41.110259 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.110242    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-520df-serving-cert\""
Jul 07 14:25:41.118988 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.118960    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"]
Jul 07 14:25:41.232417 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.232380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c77788d1-a4e8-4fda-aba3-5f7cc5558318-openshift-service-ca-bundle\") pod \"switch-graph-520df-7fcf5f5875-w848t\" (UID: \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\") " pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:41.232598 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.232426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c77788d1-a4e8-4fda-aba3-5f7cc5558318-proxy-tls\") pod \"switch-graph-520df-7fcf5f5875-w848t\" (UID: \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\") " pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:41.333200 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.333169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c77788d1-a4e8-4fda-aba3-5f7cc5558318-openshift-service-ca-bundle\") pod \"switch-graph-520df-7fcf5f5875-w848t\" (UID: \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\") " pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:41.333372 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.333210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c77788d1-a4e8-4fda-aba3-5f7cc5558318-proxy-tls\") pod \"switch-graph-520df-7fcf5f5875-w848t\" (UID: \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\") " pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:41.333876 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.333850    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c77788d1-a4e8-4fda-aba3-5f7cc5558318-openshift-service-ca-bundle\") pod \"switch-graph-520df-7fcf5f5875-w848t\" (UID: \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\") " pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:41.335554 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.335535    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c77788d1-a4e8-4fda-aba3-5f7cc5558318-proxy-tls\") pod \"switch-graph-520df-7fcf5f5875-w848t\" (UID: \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\") " pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:41.415370 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.415343    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:41.557663 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.557628    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"]
Jul 07 14:25:41.561038 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:25:41.561006    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77788d1_a4e8_4fda_aba3_5f7cc5558318.slice/crio-3894bd0f7b79bffb764c8f419c7d79c6de428767237c8fae1b0c4e0cf7762fe5 WatchSource:0}: Error finding container 3894bd0f7b79bffb764c8f419c7d79c6de428767237c8fae1b0c4e0cf7762fe5: Status 404 returned error can't find the container with id 3894bd0f7b79bffb764c8f419c7d79c6de428767237c8fae1b0c4e0cf7762fe5
Jul 07 14:25:41.945223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.945186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" event={"ID":"c77788d1-a4e8-4fda-aba3-5f7cc5558318","Type":"ContainerStarted","Data":"f0ba322a875c56563ceddb659f51fcab2c400e0d0d492868de74396f3720fbd0"}
Jul 07 14:25:41.945393 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.945241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" event={"ID":"c77788d1-a4e8-4fda-aba3-5f7cc5558318","Type":"ContainerStarted","Data":"3894bd0f7b79bffb764c8f419c7d79c6de428767237c8fae1b0c4e0cf7762fe5"}
Jul 07 14:25:41.945393 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.945307    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:41.963741 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:41.963674    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" podStartSLOduration=0.963654191 podStartE2EDuration="963.654191ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:41.963275992 +0000 UTC m=+992.960866714" watchObservedRunningTime="2026-07-07 14:25:41.963654191 +0000 UTC m=+992.961244908"
Jul 07 14:25:42.878782 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:42.878742    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:42.879225 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:42.878880    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:25:47.878688 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:47.878645    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:47.953781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:47.953753    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:25:49.924492 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:49.924450    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 14:25:52.878762 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:52.878717    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:57.878508 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:57.878466    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:25:59.924381 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:25:59.924334    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 14:26:00.950809 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:00.950782    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:26:01.015778 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.015746    2565 generic.go:356] "Generic (PLEG): container finished" podID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerID="f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23" exitCode=0
Jul 07 14:26:01.015945 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.015796    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"
Jul 07 14:26:01.015945 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.015837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" event={"ID":"7fd44486-5832-447e-a7da-ed2a1be0f4bb","Type":"ContainerDied","Data":"f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23"}
Jul 07 14:26:01.015945 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.015919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf" event={"ID":"7fd44486-5832-447e-a7da-ed2a1be0f4bb","Type":"ContainerDied","Data":"38f028abc7aabe7ab06b04cad4b81ec87df15ed18642db27092aa2800cfab3f2"}
Jul 07 14:26:01.016047 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.015946    2565 scope.go:117] "RemoveContainer" containerID="f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23"
Jul 07 14:26:01.024054 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.024034    2565 scope.go:117] "RemoveContainer" containerID="f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23"
Jul 07 14:26:01.024331 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:26:01.024315    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23\": container with ID starting with f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23 not found: ID does not exist" containerID="f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23"
Jul 07 14:26:01.024377 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.024342    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23"} err="failed to get container status \"f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23\": rpc error: code = NotFound desc = could not find container \"f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23\": container with ID starting with f832df22956eefb9cdad01b42ad1bddfaf4da1bc13b718eec0292725e205fd23 not found: ID does not exist"
Jul 07 14:26:01.094779 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.094689    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fd44486-5832-447e-a7da-ed2a1be0f4bb-proxy-tls\") pod \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\" (UID: \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\") "
Jul 07 14:26:01.094779 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.094756    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd44486-5832-447e-a7da-ed2a1be0f4bb-openshift-service-ca-bundle\") pod \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\" (UID: \"7fd44486-5832-447e-a7da-ed2a1be0f4bb\") "
Jul 07 14:26:01.095149 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.095116    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd44486-5832-447e-a7da-ed2a1be0f4bb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7fd44486-5832-447e-a7da-ed2a1be0f4bb" (UID: "7fd44486-5832-447e-a7da-ed2a1be0f4bb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:26:01.096802 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.096777    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd44486-5832-447e-a7da-ed2a1be0f4bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7fd44486-5832-447e-a7da-ed2a1be0f4bb" (UID: "7fd44486-5832-447e-a7da-ed2a1be0f4bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:26:01.195978 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.195941    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fd44486-5832-447e-a7da-ed2a1be0f4bb-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:26:01.195978 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.195971    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fd44486-5832-447e-a7da-ed2a1be0f4bb-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:26:01.337698 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.337662    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"]
Jul 07 14:26:01.340751 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.340718    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fcb978798-2vdlf"]
Jul 07 14:26:01.485509 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:01.485477    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" path="/var/lib/kubelet/pods/7fd44486-5832-447e-a7da-ed2a1be0f4bb/volumes"
Jul 07 14:26:09.924973 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:09.924931    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 14:26:19.925522 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:19.925491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:26:31.041891 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.041859    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"]
Jul 07 14:26:31.042364 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.042162    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer"
Jul 07 14:26:31.042364 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.042200    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer"
Jul 07 14:26:31.042364 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.042261    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd44486-5832-447e-a7da-ed2a1be0f4bb" containerName="model-chainer"
Jul 07 14:26:31.046277 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.046258    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:26:31.048883 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.048850    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ab362-kube-rbac-proxy-sar-config\""
Jul 07 14:26:31.048992 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.048958    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-ab362-serving-cert\""
Jul 07 14:26:31.054850 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.054828    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"]
Jul 07 14:26:31.133761 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.133730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bad35b6-a0bf-4113-bacd-c735a955dd79-openshift-service-ca-bundle\") pod \"sequence-graph-ab362-6f9b459797-kj6g5\" (UID: \"5bad35b6-a0bf-4113-bacd-c735a955dd79\") " pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:26:31.133761 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.133769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bad35b6-a0bf-4113-bacd-c735a955dd79-proxy-tls\") pod \"sequence-graph-ab362-6f9b459797-kj6g5\" (UID: \"5bad35b6-a0bf-4113-bacd-c735a955dd79\") " pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:26:31.234510 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.234467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bad35b6-a0bf-4113-bacd-c735a955dd79-openshift-service-ca-bundle\") pod \"sequence-graph-ab362-6f9b459797-kj6g5\" (UID: \"5bad35b6-a0bf-4113-bacd-c735a955dd79\") " pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:26:31.234791 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.234517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bad35b6-a0bf-4113-bacd-c735a955dd79-proxy-tls\") pod \"sequence-graph-ab362-6f9b459797-kj6g5\" (UID: \"5bad35b6-a0bf-4113-bacd-c735a955dd79\") " pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:26:31.235159 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.235137    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bad35b6-a0bf-4113-bacd-c735a955dd79-openshift-service-ca-bundle\") pod \"sequence-graph-ab362-6f9b459797-kj6g5\" (UID: \"5bad35b6-a0bf-4113-bacd-c735a955dd79\") " pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:26:31.237144 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.237122    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bad35b6-a0bf-4113-bacd-c735a955dd79-proxy-tls\") pod \"sequence-graph-ab362-6f9b459797-kj6g5\" (UID: \"5bad35b6-a0bf-4113-bacd-c735a955dd79\") " pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:26:31.357283 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.357189    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:26:31.491458 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:31.491414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"]
Jul 07 14:26:31.495681 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:26:31.495653    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bad35b6_a0bf_4113_bacd_c735a955dd79.slice/crio-d9aa3650cbbabc1b0f9d8b87d8b1e84a4feb6e5f117c58e3f7a4dc52ee5625f7 WatchSource:0}: Error finding container d9aa3650cbbabc1b0f9d8b87d8b1e84a4feb6e5f117c58e3f7a4dc52ee5625f7: Status 404 returned error can't find the container with id d9aa3650cbbabc1b0f9d8b87d8b1e84a4feb6e5f117c58e3f7a4dc52ee5625f7
Jul 07 14:26:32.135863 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:32.135820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" event={"ID":"5bad35b6-a0bf-4113-bacd-c735a955dd79","Type":"ContainerStarted","Data":"31ea7d6e0acda477d2ab9304c394ecb6372110686977703059699a6d9757cba1"}
Jul 07 14:26:32.136300 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:32.135918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" event={"ID":"5bad35b6-a0bf-4113-bacd-c735a955dd79","Type":"ContainerStarted","Data":"d9aa3650cbbabc1b0f9d8b87d8b1e84a4feb6e5f117c58e3f7a4dc52ee5625f7"}
Jul 07 14:26:32.136300 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:32.135973    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:26:32.155632 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:32.155574    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" podStartSLOduration=1.155557118 podStartE2EDuration="1.155557118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:26:32.153382086 +0000 UTC m=+1043.150972802" watchObservedRunningTime="2026-07-07 14:26:32.155557118 +0000 UTC m=+1043.153147833"
Jul 07 14:26:38.146666 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:26:38.146634    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:33:55.858058 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:55.858024    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"]
Jul 07 14:33:55.860376 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:55.858231    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df" containerID="cri-o://f0ba322a875c56563ceddb659f51fcab2c400e0d0d492868de74396f3720fbd0" gracePeriod=30
Jul 07 14:33:56.054129 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.054095    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"]
Jul 07 14:33:56.054466 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.054409    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container" containerID="cri-o://7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1" gracePeriod=30
Jul 07 14:33:56.054671 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.054608    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kube-rbac-proxy" containerID="cri-o://48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679" gracePeriod=30
Jul 07 14:33:56.066898 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.066874    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"]
Jul 07 14:33:56.070016 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.069998    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.072462 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.072441    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-03875-predictor-serving-cert\""
Jul 07 14:33:56.072553 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.072464    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-03875-kube-rbac-proxy-sar-config\""
Jul 07 14:33:56.081685 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.081655    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"]
Jul 07 14:33:56.137834 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.137793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f2v\" (UniqueName: \"kubernetes.io/projected/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-api-access-r4f2v\") pod \"success-200-isvc-03875-predictor-567dd775dc-h9f8v\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") " pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.138003 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.137846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-03875-predictor-567dd775dc-h9f8v\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") " pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.138003 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.137926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b4c350c-b103-46a3-879c-cd92be1dd15b-proxy-tls\") pod \"success-200-isvc-03875-predictor-567dd775dc-h9f8v\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") " pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.239178 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.239144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b4c350c-b103-46a3-879c-cd92be1dd15b-proxy-tls\") pod \"success-200-isvc-03875-predictor-567dd775dc-h9f8v\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") " pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.239345 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.239217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f2v\" (UniqueName: \"kubernetes.io/projected/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-api-access-r4f2v\") pod \"success-200-isvc-03875-predictor-567dd775dc-h9f8v\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") " pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.239345 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.239259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-03875-predictor-567dd775dc-h9f8v\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") " pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.239982 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.239962    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-03875-predictor-567dd775dc-h9f8v\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") " pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.241777 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.241756    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b4c350c-b103-46a3-879c-cd92be1dd15b-proxy-tls\") pod \"success-200-isvc-03875-predictor-567dd775dc-h9f8v\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") " pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.247131 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.247106    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f2v\" (UniqueName: \"kubernetes.io/projected/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-api-access-r4f2v\") pod \"success-200-isvc-03875-predictor-567dd775dc-h9f8v\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") " pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.382387 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.382337    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.518510 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.518472    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"]
Jul 07 14:33:56.521369 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:33:56.521347    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4c350c_b103_46a3_879c_cd92be1dd15b.slice/crio-3d153f40091440a145894fd1d0274fc4d1aa522d4abf1b30b9097fd65eae8b19 WatchSource:0}: Error finding container 3d153f40091440a145894fd1d0274fc4d1aa522d4abf1b30b9097fd65eae8b19: Status 404 returned error can't find the container with id 3d153f40091440a145894fd1d0274fc4d1aa522d4abf1b30b9097fd65eae8b19
Jul 07 14:33:56.830298 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.830210    2565 generic.go:356] "Generic (PLEG): container finished" podID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerID="48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679" exitCode=2
Jul 07 14:33:56.830441 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.830288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" event={"ID":"12825a42-a286-48f8-b8cf-493a38b1b7b0","Type":"ContainerDied","Data":"48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679"}
Jul 07 14:33:56.831927 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.831906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" event={"ID":"8b4c350c-b103-46a3-879c-cd92be1dd15b","Type":"ContainerStarted","Data":"91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e"}
Jul 07 14:33:56.832030 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.831956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" event={"ID":"8b4c350c-b103-46a3-879c-cd92be1dd15b","Type":"ContainerStarted","Data":"8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67"}
Jul 07 14:33:56.832030 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.831977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" event={"ID":"8b4c350c-b103-46a3-879c-cd92be1dd15b","Type":"ContainerStarted","Data":"3d153f40091440a145894fd1d0274fc4d1aa522d4abf1b30b9097fd65eae8b19"}
Jul 07 14:33:56.832128 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.832094    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.832263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.832233    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:33:56.833388 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.833368    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:33:56.851734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:56.851682    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podStartSLOduration=0.851666278 podStartE2EDuration="851.666278ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:56.850823528 +0000 UTC m=+1487.848414244" watchObservedRunningTime="2026-07-07 14:33:56.851666278 +0000 UTC m=+1487.849256993"
Jul 07 14:33:57.676121 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:57.676077    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 07 14:33:57.682515 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:57.682483    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 14:33:57.840529 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:57.839031    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:33:57.952885 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:57.952780    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:33:59.507741 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.507717    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:33:59.565079 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.565000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-rbac-proxy-sar-config\") pod \"12825a42-a286-48f8-b8cf-493a38b1b7b0\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") "
Jul 07 14:33:59.565079 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.565042    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj62n\" (UniqueName: \"kubernetes.io/projected/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-api-access-lj62n\") pod \"12825a42-a286-48f8-b8cf-493a38b1b7b0\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") "
Jul 07 14:33:59.565079 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.565076    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12825a42-a286-48f8-b8cf-493a38b1b7b0-proxy-tls\") pod \"12825a42-a286-48f8-b8cf-493a38b1b7b0\" (UID: \"12825a42-a286-48f8-b8cf-493a38b1b7b0\") "
Jul 07 14:33:59.565541 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.565393    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12825a42-a286-48f8-b8cf-493a38b1b7b0" (UID: "12825a42-a286-48f8-b8cf-493a38b1b7b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:33:59.567227 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.567206    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-api-access-lj62n" (OuterVolumeSpecName: "kube-api-access-lj62n") pod "12825a42-a286-48f8-b8cf-493a38b1b7b0" (UID: "12825a42-a286-48f8-b8cf-493a38b1b7b0"). InnerVolumeSpecName "kube-api-access-lj62n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:33:59.567337 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.567313    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12825a42-a286-48f8-b8cf-493a38b1b7b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12825a42-a286-48f8-b8cf-493a38b1b7b0" (UID: "12825a42-a286-48f8-b8cf-493a38b1b7b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:33:59.665902 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.665866    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:33:59.665902 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.665895    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj62n\" (UniqueName: \"kubernetes.io/projected/12825a42-a286-48f8-b8cf-493a38b1b7b0-kube-api-access-lj62n\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:33:59.665902 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.665905    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12825a42-a286-48f8-b8cf-493a38b1b7b0-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:33:59.846717 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.846602    2565 generic.go:356] "Generic (PLEG): container finished" podID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerID="7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1" exitCode=0
Jul 07 14:33:59.846717 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.846686    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"
Jul 07 14:33:59.846717 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.846690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" event={"ID":"12825a42-a286-48f8-b8cf-493a38b1b7b0","Type":"ContainerDied","Data":"7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1"}
Jul 07 14:33:59.846919 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.846763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm" event={"ID":"12825a42-a286-48f8-b8cf-493a38b1b7b0","Type":"ContainerDied","Data":"254ba14939212566b076295a23b9efbbf49f019054175ed038ff4bbac9c2ed37"}
Jul 07 14:33:59.846919 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.846801    2565 scope.go:117] "RemoveContainer" containerID="48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679"
Jul 07 14:33:59.855029 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.855012    2565 scope.go:117] "RemoveContainer" containerID="7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1"
Jul 07 14:33:59.862304 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.862288    2565 scope.go:117] "RemoveContainer" containerID="48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679"
Jul 07 14:33:59.862541 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:33:59.862523    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679\": container with ID starting with 48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679 not found: ID does not exist" containerID="48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679"
Jul 07 14:33:59.862589 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.862548    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679"} err="failed to get container status \"48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679\": rpc error: code = NotFound desc = could not find container \"48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679\": container with ID starting with 48043ebb2621c5e7182c85f51d2c26eb354f2503a85c3d489a69d078e8bc6679 not found: ID does not exist"
Jul 07 14:33:59.862589 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.862564    2565 scope.go:117] "RemoveContainer" containerID="7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1"
Jul 07 14:33:59.862801 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:33:59.862781    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1\": container with ID starting with 7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1 not found: ID does not exist" containerID="7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1"
Jul 07 14:33:59.862855 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.862808    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1"} err="failed to get container status \"7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1\": rpc error: code = NotFound desc = could not find container \"7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1\": container with ID starting with 7a17ed68605cc95ff67d48e5145bb4ca7aadeae74737f873103a14903fe223e1 not found: ID does not exist"
Jul 07 14:33:59.867706 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.867675    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"]
Jul 07 14:33:59.875929 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:33:59.875898    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-520df-predictor-86849fb99-w5khm"]
Jul 07 14:34:01.486177 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:01.486144    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" path="/var/lib/kubelet/pods/12825a42-a286-48f8-b8cf-493a38b1b7b0/volumes"
Jul 07 14:34:02.843876 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:02.843841    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:34:02.844382 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:02.844351    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:34:02.952907 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:02.952866    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:34:07.952566 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:07.952528    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:34:07.952977 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:07.952674    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:34:12.844687 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:12.844640    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:34:12.952575 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:12.952534    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:34:17.952797 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:17.952752    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:34:22.845115 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:22.845072    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:34:22.952154 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:22.952110    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:34:25.952148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:25.952101    2565 generic.go:356] "Generic (PLEG): container finished" podID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerID="f0ba322a875c56563ceddb659f51fcab2c400e0d0d492868de74396f3720fbd0" exitCode=0
Jul 07 14:34:25.952508 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:25.952174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" event={"ID":"c77788d1-a4e8-4fda-aba3-5f7cc5558318","Type":"ContainerDied","Data":"f0ba322a875c56563ceddb659f51fcab2c400e0d0d492868de74396f3720fbd0"}
Jul 07 14:34:26.057550 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.057526    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:34:26.183163 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.183132    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c77788d1-a4e8-4fda-aba3-5f7cc5558318-proxy-tls\") pod \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\" (UID: \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\") "
Jul 07 14:34:26.183331 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.183223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c77788d1-a4e8-4fda-aba3-5f7cc5558318-openshift-service-ca-bundle\") pod \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\" (UID: \"c77788d1-a4e8-4fda-aba3-5f7cc5558318\") "
Jul 07 14:34:26.183583 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.183559    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77788d1-a4e8-4fda-aba3-5f7cc5558318-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c77788d1-a4e8-4fda-aba3-5f7cc5558318" (UID: "c77788d1-a4e8-4fda-aba3-5f7cc5558318"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:34:26.185294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.185271    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77788d1-a4e8-4fda-aba3-5f7cc5558318-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c77788d1-a4e8-4fda-aba3-5f7cc5558318" (UID: "c77788d1-a4e8-4fda-aba3-5f7cc5558318"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:34:26.284366 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.284316    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c77788d1-a4e8-4fda-aba3-5f7cc5558318-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:34:26.284366 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.284366    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c77788d1-a4e8-4fda-aba3-5f7cc5558318-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:34:26.956860 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.956833    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"
Jul 07 14:34:26.957309 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.956836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t" event={"ID":"c77788d1-a4e8-4fda-aba3-5f7cc5558318","Type":"ContainerDied","Data":"3894bd0f7b79bffb764c8f419c7d79c6de428767237c8fae1b0c4e0cf7762fe5"}
Jul 07 14:34:26.957309 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.956966    2565 scope.go:117] "RemoveContainer" containerID="f0ba322a875c56563ceddb659f51fcab2c400e0d0d492868de74396f3720fbd0"
Jul 07 14:34:26.978501 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.978470    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"]
Jul 07 14:34:26.982286 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:26.982257    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-520df-7fcf5f5875-w848t"]
Jul 07 14:34:27.486873 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:27.486843    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" path="/var/lib/kubelet/pods/c77788d1-a4e8-4fda-aba3-5f7cc5558318/volumes"
Jul 07 14:34:32.844773 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:32.844681    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 14:34:42.844779 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:42.844746    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:34:45.602782 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.602745    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"]
Jul 07 14:34:45.735073 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.735039    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"]
Jul 07 14:34:45.735351 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.735275    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container" containerID="cri-o://18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1" gracePeriod=30
Jul 07 14:34:45.735351 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.735320    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kube-rbac-proxy" containerID="cri-o://47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4" gracePeriod=30
Jul 07 14:34:45.756905 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.756876    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"]
Jul 07 14:34:45.757223 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.757207    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kube-rbac-proxy"
Jul 07 14:34:45.757263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.757228    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kube-rbac-proxy"
Jul 07 14:34:45.757263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.757248    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df"
Jul 07 14:34:45.757263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.757260    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df"
Jul 07 14:34:45.757351 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.757275    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container"
Jul 07 14:34:45.757351 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.757283    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container"
Jul 07 14:34:45.757414 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.757360    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kube-rbac-proxy"
Jul 07 14:34:45.757414 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.757376    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="12825a42-a286-48f8-b8cf-493a38b1b7b0" containerName="kserve-container"
Jul 07 14:34:45.757414 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.757383    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c77788d1-a4e8-4fda-aba3-5f7cc5558318" containerName="switch-graph-520df"
Jul 07 14:34:45.760354 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.760335    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:45.762472 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.762444    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fbc26-predictor-serving-cert\""
Jul 07 14:34:45.762472 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.762463    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fbc26-kube-rbac-proxy-sar-config\""
Jul 07 14:34:45.771137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.771109    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"]
Jul 07 14:34:45.837438 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.837394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd4bd77b-9822-4829-8e54-d6e24b6f264c-proxy-tls\") pod \"success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:45.837656 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.837452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:45.837656 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.837514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9cc\" (UniqueName: \"kubernetes.io/projected/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-api-access-4n9cc\") pod \"success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:45.938769 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.938737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd4bd77b-9822-4829-8e54-d6e24b6f264c-proxy-tls\") pod \"success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:45.938961 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.938795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:45.938961 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.938823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9cc\" (UniqueName: \"kubernetes.io/projected/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-api-access-4n9cc\") pod \"success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:45.939648 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.939603    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:45.941468 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.941440    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd4bd77b-9822-4829-8e54-d6e24b6f264c-proxy-tls\") pod \"success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:45.946314 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:45.946285    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9cc\" (UniqueName: \"kubernetes.io/projected/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-api-access-4n9cc\") pod \"success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:46.032388 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:46.032358    2565 generic.go:356] "Generic (PLEG): container finished" podID="83654928-fe75-4138-90ed-3acf87dad1ae" containerID="47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4" exitCode=2
Jul 07 14:34:46.032553 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:46.032430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" event={"ID":"83654928-fe75-4138-90ed-3acf87dad1ae","Type":"ContainerDied","Data":"47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4"}
Jul 07 14:34:46.032599 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:46.032550    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362" containerID="cri-o://31ea7d6e0acda477d2ab9304c394ecb6372110686977703059699a6d9757cba1" gracePeriod=30
Jul 07 14:34:46.070588 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:46.070546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:46.210715 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:46.210690    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"]
Jul 07 14:34:46.213171 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:34:46.213138    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4bd77b_9822_4829_8e54_d6e24b6f264c.slice/crio-8d2ed63fbb3ac57f73f6916df116325d5d8cac173b44d1c095e9fecb28a0f906 WatchSource:0}: Error finding container 8d2ed63fbb3ac57f73f6916df116325d5d8cac173b44d1c095e9fecb28a0f906: Status 404 returned error can't find the container with id 8d2ed63fbb3ac57f73f6916df116325d5d8cac173b44d1c095e9fecb28a0f906
Jul 07 14:34:47.040199 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:47.040163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" event={"ID":"fd4bd77b-9822-4829-8e54-d6e24b6f264c","Type":"ContainerStarted","Data":"9af9146840712ee315b302afd0039cf39bee54a834de8e5c0bb3ecbfb74fa60f"}
Jul 07 14:34:47.040569 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:47.040221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" event={"ID":"fd4bd77b-9822-4829-8e54-d6e24b6f264c","Type":"ContainerStarted","Data":"16d93bba4583a2046528290364404e2e190709555d969da6a13d0139351afa93"}
Jul 07 14:34:47.040569 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:47.040247    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:47.040569 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:47.040269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" event={"ID":"fd4bd77b-9822-4829-8e54-d6e24b6f264c","Type":"ContainerStarted","Data":"8d2ed63fbb3ac57f73f6916df116325d5d8cac173b44d1c095e9fecb28a0f906"}
Jul 07 14:34:47.063750 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:47.063685    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podStartSLOduration=2.063667824 podStartE2EDuration="2.063667824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:47.062318398 +0000 UTC m=+1538.059909125" watchObservedRunningTime="2026-07-07 14:34:47.063667824 +0000 UTC m=+1538.061258538"
Jul 07 14:34:48.044007 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:48.043972    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:48.045454 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:48.045424    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:34:48.144269 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:48.144231    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:34:49.047976 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.047937    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:34:49.187449 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.187423    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:34:49.266258 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.266148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqrjq\" (UniqueName: \"kubernetes.io/projected/83654928-fe75-4138-90ed-3acf87dad1ae-kube-api-access-qqrjq\") pod \"83654928-fe75-4138-90ed-3acf87dad1ae\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") "
Jul 07 14:34:49.266258 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.266206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83654928-fe75-4138-90ed-3acf87dad1ae-kube-rbac-proxy-sar-config\") pod \"83654928-fe75-4138-90ed-3acf87dad1ae\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") "
Jul 07 14:34:49.266258 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.266257    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83654928-fe75-4138-90ed-3acf87dad1ae-proxy-tls\") pod \"83654928-fe75-4138-90ed-3acf87dad1ae\" (UID: \"83654928-fe75-4138-90ed-3acf87dad1ae\") "
Jul 07 14:34:49.266606 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.266571    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83654928-fe75-4138-90ed-3acf87dad1ae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83654928-fe75-4138-90ed-3acf87dad1ae" (UID: "83654928-fe75-4138-90ed-3acf87dad1ae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:34:49.268309 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.268288    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83654928-fe75-4138-90ed-3acf87dad1ae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83654928-fe75-4138-90ed-3acf87dad1ae" (UID: "83654928-fe75-4138-90ed-3acf87dad1ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:34:49.268408 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.268391    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83654928-fe75-4138-90ed-3acf87dad1ae-kube-api-access-qqrjq" (OuterVolumeSpecName: "kube-api-access-qqrjq") pod "83654928-fe75-4138-90ed-3acf87dad1ae" (UID: "83654928-fe75-4138-90ed-3acf87dad1ae"). InnerVolumeSpecName "kube-api-access-qqrjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:34:49.367219 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.367178    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqrjq\" (UniqueName: \"kubernetes.io/projected/83654928-fe75-4138-90ed-3acf87dad1ae-kube-api-access-qqrjq\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:34:49.367219 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.367207    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83654928-fe75-4138-90ed-3acf87dad1ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:34:49.367219 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:49.367219    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83654928-fe75-4138-90ed-3acf87dad1ae-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:34:50.053678 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.053638    2565 generic.go:356] "Generic (PLEG): container finished" podID="83654928-fe75-4138-90ed-3acf87dad1ae" containerID="18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1" exitCode=0
Jul 07 14:34:50.054063 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.053698    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"
Jul 07 14:34:50.054063 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.053714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" event={"ID":"83654928-fe75-4138-90ed-3acf87dad1ae","Type":"ContainerDied","Data":"18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1"}
Jul 07 14:34:50.054063 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.053800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45" event={"ID":"83654928-fe75-4138-90ed-3acf87dad1ae","Type":"ContainerDied","Data":"53a98d0e292cbe1adf5c7490791a8185a193ececc17559f97d86908792c9ac3e"}
Jul 07 14:34:50.054063 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.053826    2565 scope.go:117] "RemoveContainer" containerID="47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4"
Jul 07 14:34:50.061922 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.061902    2565 scope.go:117] "RemoveContainer" containerID="18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1"
Jul 07 14:34:50.069600 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.069575    2565 scope.go:117] "RemoveContainer" containerID="47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4"
Jul 07 14:34:50.070140 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:34:50.070114    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4\": container with ID starting with 47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4 not found: ID does not exist" containerID="47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4"
Jul 07 14:34:50.070249 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.070150    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4"} err="failed to get container status \"47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4\": rpc error: code = NotFound desc = could not find container \"47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4\": container with ID starting with 47e7ba797ba3d290e37d0b07a5ef4346c4385daad4504843c9323ccde15597f4 not found: ID does not exist"
Jul 07 14:34:50.070249 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.070176    2565 scope.go:117] "RemoveContainer" containerID="18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1"
Jul 07 14:34:50.070455 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:34:50.070433    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1\": container with ID starting with 18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1 not found: ID does not exist" containerID="18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1"
Jul 07 14:34:50.070496 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.070467    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1"} err="failed to get container status \"18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1\": rpc error: code = NotFound desc = could not find container \"18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1\": container with ID starting with 18b32ea5218af3684be73e198551ae6cd68af126ab55cd84edbd28982f4782e1 not found: ID does not exist"
Jul 07 14:34:50.071189 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.071172    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"]
Jul 07 14:34:50.074346 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:50.074326    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab362-predictor-5c95c8c5d9-bdd45"]
Jul 07 14:34:51.485898 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:51.485855    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" path="/var/lib/kubelet/pods/83654928-fe75-4138-90ed-3acf87dad1ae/volumes"
Jul 07 14:34:53.144550 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:53.144505    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:34:54.053172 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:54.053142    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:34:54.053817 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:54.053788    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:34:56.101934 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.101896    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"]
Jul 07 14:34:56.102355 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.102186    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container"
Jul 07 14:34:56.102355 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.102197    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container"
Jul 07 14:34:56.102355 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.102216    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kube-rbac-proxy"
Jul 07 14:34:56.102355 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.102222    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kube-rbac-proxy"
Jul 07 14:34:56.102355 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.102274    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kserve-container"
Jul 07 14:34:56.102355 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.102283    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="83654928-fe75-4138-90ed-3acf87dad1ae" containerName="kube-rbac-proxy"
Jul 07 14:34:56.106835 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.106815    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:34:56.109171 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.109151    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-03875-kube-rbac-proxy-sar-config\""
Jul 07 14:34:56.109287 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.109170    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-03875-serving-cert\""
Jul 07 14:34:56.115341 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.115316    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"]
Jul 07 14:34:56.221798 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.221756    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54488bfb-5bd8-4cf2-af48-f23398ddda74-proxy-tls\") pod \"ensemble-graph-03875-56c5b76d48-sbw5h\" (UID: \"54488bfb-5bd8-4cf2-af48-f23398ddda74\") " pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:34:56.221950 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.221840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54488bfb-5bd8-4cf2-af48-f23398ddda74-openshift-service-ca-bundle\") pod \"ensemble-graph-03875-56c5b76d48-sbw5h\" (UID: \"54488bfb-5bd8-4cf2-af48-f23398ddda74\") " pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:34:56.322261 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.322219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54488bfb-5bd8-4cf2-af48-f23398ddda74-openshift-service-ca-bundle\") pod \"ensemble-graph-03875-56c5b76d48-sbw5h\" (UID: \"54488bfb-5bd8-4cf2-af48-f23398ddda74\") " pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:34:56.322440 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.322277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54488bfb-5bd8-4cf2-af48-f23398ddda74-proxy-tls\") pod \"ensemble-graph-03875-56c5b76d48-sbw5h\" (UID: \"54488bfb-5bd8-4cf2-af48-f23398ddda74\") " pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:34:56.322908 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.322878    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54488bfb-5bd8-4cf2-af48-f23398ddda74-openshift-service-ca-bundle\") pod \"ensemble-graph-03875-56c5b76d48-sbw5h\" (UID: \"54488bfb-5bd8-4cf2-af48-f23398ddda74\") " pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:34:56.324835 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.324814    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54488bfb-5bd8-4cf2-af48-f23398ddda74-proxy-tls\") pod \"ensemble-graph-03875-56c5b76d48-sbw5h\" (UID: \"54488bfb-5bd8-4cf2-af48-f23398ddda74\") " pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:34:56.417665 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.417605    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:34:56.553918 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:56.553881    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"]
Jul 07 14:34:57.084168 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:57.084124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" event={"ID":"54488bfb-5bd8-4cf2-af48-f23398ddda74","Type":"ContainerStarted","Data":"9f13c3378a076116cbc12fb3b201bfe8c612df79fcdb66afdf1bdc32b1476ce3"}
Jul 07 14:34:57.084344 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:57.084228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" event={"ID":"54488bfb-5bd8-4cf2-af48-f23398ddda74","Type":"ContainerStarted","Data":"fe874bf7f05abe22fe9a1c71d0d6a915cb13b06d6eae9ed0a3ad2ab96b2513bf"}
Jul 07 14:34:57.084344 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:57.084260    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:34:57.103361 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:57.103313    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" podStartSLOduration=1.103296324 podStartE2EDuration="1.103296324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:57.102122745 +0000 UTC m=+1548.099713459" watchObservedRunningTime="2026-07-07 14:34:57.103296324 +0000 UTC m=+1548.100887039"
Jul 07 14:34:58.144560 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:58.144522    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:34:58.145041 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:34:58.144659    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:35:03.093910 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:03.093873    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:35:03.144209 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:03.144170    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:04.053957 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:04.053911    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:35:06.174552 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.174518    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"]
Jul 07 14:35:06.175019 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.174733    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875" containerID="cri-o://9f13c3378a076116cbc12fb3b201bfe8c612df79fcdb66afdf1bdc32b1476ce3" gracePeriod=30
Jul 07 14:35:06.305678 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.305642    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"]
Jul 07 14:35:06.305937 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.305906    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container" containerID="cri-o://8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67" gracePeriod=30
Jul 07 14:35:06.305937 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.305921    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kube-rbac-proxy" containerID="cri-o://91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e" gracePeriod=30
Jul 07 14:35:06.309324 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.309298    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"]
Jul 07 14:35:06.313326 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.313306    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:06.315465 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.315439    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fe15b-predictor-serving-cert\""
Jul 07 14:35:06.315555 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.315536    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fe15b-kube-rbac-proxy-sar-config\""
Jul 07 14:35:06.327765 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.327740    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"]
Jul 07 14:35:06.406416 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.406386    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963cf95d-8dda-4225-b8e7-d945024f7253-proxy-tls\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:06.406585 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.406433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963cf95d-8dda-4225-b8e7-d945024f7253-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:06.406585 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.406509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbtb8\" (UniqueName: \"kubernetes.io/projected/963cf95d-8dda-4225-b8e7-d945024f7253-kube-api-access-kbtb8\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:06.507947 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.507856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963cf95d-8dda-4225-b8e7-d945024f7253-proxy-tls\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:06.507947 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.507908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963cf95d-8dda-4225-b8e7-d945024f7253-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:06.508190 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.507954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtb8\" (UniqueName: \"kubernetes.io/projected/963cf95d-8dda-4225-b8e7-d945024f7253-kube-api-access-kbtb8\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:06.508190 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:35:06.508013    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-serving-cert: secret "success-200-isvc-fe15b-predictor-serving-cert" not found
Jul 07 14:35:06.508190 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:35:06.508088    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/963cf95d-8dda-4225-b8e7-d945024f7253-proxy-tls podName:963cf95d-8dda-4225-b8e7-d945024f7253 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:07.008071426 +0000 UTC m=+1558.005662119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/963cf95d-8dda-4225-b8e7-d945024f7253-proxy-tls") pod "success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" (UID: "963cf95d-8dda-4225-b8e7-d945024f7253") : secret "success-200-isvc-fe15b-predictor-serving-cert" not found
Jul 07 14:35:06.508696 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.508670    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963cf95d-8dda-4225-b8e7-d945024f7253-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:06.517688 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:06.517663    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtb8\" (UniqueName: \"kubernetes.io/projected/963cf95d-8dda-4225-b8e7-d945024f7253-kube-api-access-kbtb8\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:07.011648 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:07.011587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963cf95d-8dda-4225-b8e7-d945024f7253-proxy-tls\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:07.014044 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:07.013997    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963cf95d-8dda-4225-b8e7-d945024f7253-proxy-tls\") pod \"success-200-isvc-fe15b-predictor-6654f94d55-dmfx4\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") " pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:07.122811 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:07.122777    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerID="91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e" exitCode=2
Jul 07 14:35:07.122988 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:07.122851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" event={"ID":"8b4c350c-b103-46a3-879c-cd92be1dd15b","Type":"ContainerDied","Data":"91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e"}
Jul 07 14:35:07.228325 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:07.228279    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:07.368497 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:07.368436    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"]
Jul 07 14:35:07.372180 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:35:07.372138    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963cf95d_8dda_4225_b8e7_d945024f7253.slice/crio-c75bae53f698dfb11a8cb3a82fd1e5dcb44bfc72a82797f0eeae55ade87e3778 WatchSource:0}: Error finding container c75bae53f698dfb11a8cb3a82fd1e5dcb44bfc72a82797f0eeae55ade87e3778: Status 404 returned error can't find the container with id c75bae53f698dfb11a8cb3a82fd1e5dcb44bfc72a82797f0eeae55ade87e3778
Jul 07 14:35:07.840046 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:07.840002    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 14:35:08.093158 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:08.093071    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:08.128355 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:08.128316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" event={"ID":"963cf95d-8dda-4225-b8e7-d945024f7253","Type":"ContainerStarted","Data":"bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435"}
Jul 07 14:35:08.128509 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:08.128384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" event={"ID":"963cf95d-8dda-4225-b8e7-d945024f7253","Type":"ContainerStarted","Data":"7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2"}
Jul 07 14:35:08.128509 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:08.128412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" event={"ID":"963cf95d-8dda-4225-b8e7-d945024f7253","Type":"ContainerStarted","Data":"c75bae53f698dfb11a8cb3a82fd1e5dcb44bfc72a82797f0eeae55ade87e3778"}
Jul 07 14:35:08.128509 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:08.128474    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:08.143807 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:08.143772    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:08.152804 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:08.152756    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" podStartSLOduration=2.152740293 podStartE2EDuration="2.152740293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:08.151233415 +0000 UTC m=+1559.148824131" watchObservedRunningTime="2026-07-07 14:35:08.152740293 +0000 UTC m=+1559.150331008"
Jul 07 14:35:09.133264 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.133225    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:09.134507 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.134478    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:35:09.655306 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.655282    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:35:09.735705 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.735581    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b4c350c-b103-46a3-879c-cd92be1dd15b-proxy-tls\") pod \"8b4c350c-b103-46a3-879c-cd92be1dd15b\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") "
Jul 07 14:35:09.735705 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.735682    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-rbac-proxy-sar-config\") pod \"8b4c350c-b103-46a3-879c-cd92be1dd15b\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") "
Jul 07 14:35:09.735927 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.735715    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4f2v\" (UniqueName: \"kubernetes.io/projected/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-api-access-r4f2v\") pod \"8b4c350c-b103-46a3-879c-cd92be1dd15b\" (UID: \"8b4c350c-b103-46a3-879c-cd92be1dd15b\") "
Jul 07 14:35:09.736070 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.736046    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b4c350c-b103-46a3-879c-cd92be1dd15b" (UID: "8b4c350c-b103-46a3-879c-cd92be1dd15b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:35:09.738026 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.737975    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-api-access-r4f2v" (OuterVolumeSpecName: "kube-api-access-r4f2v") pod "8b4c350c-b103-46a3-879c-cd92be1dd15b" (UID: "8b4c350c-b103-46a3-879c-cd92be1dd15b"). InnerVolumeSpecName "kube-api-access-r4f2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:35:09.738026 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.737985    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4c350c-b103-46a3-879c-cd92be1dd15b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b4c350c-b103-46a3-879c-cd92be1dd15b" (UID: "8b4c350c-b103-46a3-879c-cd92be1dd15b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:35:09.836782 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.836741    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:35:09.836782 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.836776    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f2v\" (UniqueName: \"kubernetes.io/projected/8b4c350c-b103-46a3-879c-cd92be1dd15b-kube-api-access-r4f2v\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:35:09.836988 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:09.836790    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b4c350c-b103-46a3-879c-cd92be1dd15b-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:35:10.139034 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.139001    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerID="8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67" exitCode=0
Jul 07 14:35:10.139463 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.139050    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"
Jul 07 14:35:10.139463 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.139093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" event={"ID":"8b4c350c-b103-46a3-879c-cd92be1dd15b","Type":"ContainerDied","Data":"8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67"}
Jul 07 14:35:10.139463 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.139149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v" event={"ID":"8b4c350c-b103-46a3-879c-cd92be1dd15b","Type":"ContainerDied","Data":"3d153f40091440a145894fd1d0274fc4d1aa522d4abf1b30b9097fd65eae8b19"}
Jul 07 14:35:10.139463 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.139181    2565 scope.go:117] "RemoveContainer" containerID="91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e"
Jul 07 14:35:10.139675 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.139592    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:35:10.148008 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.147992    2565 scope.go:117] "RemoveContainer" containerID="8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67"
Jul 07 14:35:10.155258 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.155233    2565 scope.go:117] "RemoveContainer" containerID="91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e"
Jul 07 14:35:10.155504 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:35:10.155486    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e\": container with ID starting with 91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e not found: ID does not exist" containerID="91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e"
Jul 07 14:35:10.155560 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.155515    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e"} err="failed to get container status \"91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e\": rpc error: code = NotFound desc = could not find container \"91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e\": container with ID starting with 91a92777feeeea0c609663045e3c4739aa86f3b2dc219c69d5e574e0767b0c5e not found: ID does not exist"
Jul 07 14:35:10.155560 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.155533    2565 scope.go:117] "RemoveContainer" containerID="8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67"
Jul 07 14:35:10.155785 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:35:10.155769    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67\": container with ID starting with 8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67 not found: ID does not exist" containerID="8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67"
Jul 07 14:35:10.155827 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.155789    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67"} err="failed to get container status \"8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67\": rpc error: code = NotFound desc = could not find container \"8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67\": container with ID starting with 8d767679f48ef9535457a22036476c6abed1c1f0c8f15699c89fba972cf09e67 not found: ID does not exist"
Jul 07 14:35:10.160858 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.160832    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"]
Jul 07 14:35:10.166300 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:10.166267    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-03875-predictor-567dd775dc-h9f8v"]
Jul 07 14:35:11.486709 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:11.486678    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" path="/var/lib/kubelet/pods/8b4c350c-b103-46a3-879c-cd92be1dd15b/volumes"
Jul 07 14:35:13.092484 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:13.092440    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:13.144159 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:13.144123    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:14.054385 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:14.054344    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:35:15.144502 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:15.144475    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:15.145004 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:15.144980    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:35:16.165397 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:16.165367    2565 generic.go:356] "Generic (PLEG): container finished" podID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerID="31ea7d6e0acda477d2ab9304c394ecb6372110686977703059699a6d9757cba1" exitCode=0
Jul 07 14:35:16.165743 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:16.165442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" event={"ID":"5bad35b6-a0bf-4113-bacd-c735a955dd79","Type":"ContainerDied","Data":"31ea7d6e0acda477d2ab9304c394ecb6372110686977703059699a6d9757cba1"}
Jul 07 14:35:16.177953 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:16.177929    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:35:16.290946 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:16.290855    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bad35b6-a0bf-4113-bacd-c735a955dd79-proxy-tls\") pod \"5bad35b6-a0bf-4113-bacd-c735a955dd79\" (UID: \"5bad35b6-a0bf-4113-bacd-c735a955dd79\") "
Jul 07 14:35:16.290946 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:16.290903    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bad35b6-a0bf-4113-bacd-c735a955dd79-openshift-service-ca-bundle\") pod \"5bad35b6-a0bf-4113-bacd-c735a955dd79\" (UID: \"5bad35b6-a0bf-4113-bacd-c735a955dd79\") "
Jul 07 14:35:16.291324 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:16.291298    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bad35b6-a0bf-4113-bacd-c735a955dd79-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5bad35b6-a0bf-4113-bacd-c735a955dd79" (UID: "5bad35b6-a0bf-4113-bacd-c735a955dd79"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:35:16.293120 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:16.293091    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bad35b6-a0bf-4113-bacd-c735a955dd79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bad35b6-a0bf-4113-bacd-c735a955dd79" (UID: "5bad35b6-a0bf-4113-bacd-c735a955dd79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:35:16.391570 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:16.391528    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bad35b6-a0bf-4113-bacd-c735a955dd79-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:35:16.391570 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:16.391560    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bad35b6-a0bf-4113-bacd-c735a955dd79-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:35:17.170126 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:17.170093    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"
Jul 07 14:35:17.170659 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:17.170087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5" event={"ID":"5bad35b6-a0bf-4113-bacd-c735a955dd79","Type":"ContainerDied","Data":"d9aa3650cbbabc1b0f9d8b87d8b1e84a4feb6e5f117c58e3f7a4dc52ee5625f7"}
Jul 07 14:35:17.170659 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:17.170239    2565 scope.go:117] "RemoveContainer" containerID="31ea7d6e0acda477d2ab9304c394ecb6372110686977703059699a6d9757cba1"
Jul 07 14:35:17.191762 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:17.191731    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"]
Jul 07 14:35:17.196515 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:17.196479    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-ab362-6f9b459797-kj6g5"]
Jul 07 14:35:17.486389 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:17.486308    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" path="/var/lib/kubelet/pods/5bad35b6-a0bf-4113-bacd-c735a955dd79/volumes"
Jul 07 14:35:18.093127 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:18.093086    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:18.093302 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:18.093201    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:35:23.093221 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:23.093185    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:24.054677 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:24.054635    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 14:35:25.145432 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:25.145395    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:35:28.092410 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:28.092368    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:33.093195 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:33.093148    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:34.054412 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:34.054384    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:35:35.145101 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:35.145063    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:35:36.245985 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:36.245946    2565 generic.go:356] "Generic (PLEG): container finished" podID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerID="9f13c3378a076116cbc12fb3b201bfe8c612df79fcdb66afdf1bdc32b1476ce3" exitCode=0
Jul 07 14:35:36.246335 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:36.246030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" event={"ID":"54488bfb-5bd8-4cf2-af48-f23398ddda74","Type":"ContainerDied","Data":"9f13c3378a076116cbc12fb3b201bfe8c612df79fcdb66afdf1bdc32b1476ce3"}
Jul 07 14:35:36.326903 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:36.326871    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:35:36.451954 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:36.451866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54488bfb-5bd8-4cf2-af48-f23398ddda74-openshift-service-ca-bundle\") pod \"54488bfb-5bd8-4cf2-af48-f23398ddda74\" (UID: \"54488bfb-5bd8-4cf2-af48-f23398ddda74\") "
Jul 07 14:35:36.451954 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:36.451923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54488bfb-5bd8-4cf2-af48-f23398ddda74-proxy-tls\") pod \"54488bfb-5bd8-4cf2-af48-f23398ddda74\" (UID: \"54488bfb-5bd8-4cf2-af48-f23398ddda74\") "
Jul 07 14:35:36.452256 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:36.452223    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54488bfb-5bd8-4cf2-af48-f23398ddda74-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "54488bfb-5bd8-4cf2-af48-f23398ddda74" (UID: "54488bfb-5bd8-4cf2-af48-f23398ddda74"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:35:36.454055 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:36.454035    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54488bfb-5bd8-4cf2-af48-f23398ddda74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54488bfb-5bd8-4cf2-af48-f23398ddda74" (UID: "54488bfb-5bd8-4cf2-af48-f23398ddda74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:35:36.553471 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:36.553427    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54488bfb-5bd8-4cf2-af48-f23398ddda74-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:35:36.553471 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:36.553461    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54488bfb-5bd8-4cf2-af48-f23398ddda74-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:35:37.251421 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:37.251393    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"
Jul 07 14:35:37.251421 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:37.251402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h" event={"ID":"54488bfb-5bd8-4cf2-af48-f23398ddda74","Type":"ContainerDied","Data":"fe874bf7f05abe22fe9a1c71d0d6a915cb13b06d6eae9ed0a3ad2ab96b2513bf"}
Jul 07 14:35:37.251917 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:37.251458    2565 scope.go:117] "RemoveContainer" containerID="9f13c3378a076116cbc12fb3b201bfe8c612df79fcdb66afdf1bdc32b1476ce3"
Jul 07 14:35:37.276365 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:37.276334    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"]
Jul 07 14:35:37.281267 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:37.281232    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-03875-56c5b76d48-sbw5h"]
Jul 07 14:35:37.486452 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:37.486420    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" path="/var/lib/kubelet/pods/54488bfb-5bd8-4cf2-af48-f23398ddda74/volumes"
Jul 07 14:35:45.145641 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.145575    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 07 14:35:45.783203 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783160    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"]
Jul 07 14:35:45.783584 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783563    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362"
Jul 07 14:35:45.783584 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783585    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362"
Jul 07 14:35:45.783781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783606    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container"
Jul 07 14:35:45.783781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783630    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container"
Jul 07 14:35:45.783781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783639    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875"
Jul 07 14:35:45.783781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783648    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875"
Jul 07 14:35:45.783781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783660    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kube-rbac-proxy"
Jul 07 14:35:45.783781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783668    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kube-rbac-proxy"
Jul 07 14:35:45.783781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783755    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kserve-container"
Jul 07 14:35:45.783781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783770    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b4c350c-b103-46a3-879c-cd92be1dd15b" containerName="kube-rbac-proxy"
Jul 07 14:35:45.783781 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783780    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="54488bfb-5bd8-4cf2-af48-f23398ddda74" containerName="ensemble-graph-03875"
Jul 07 14:35:45.784192 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.783791    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bad35b6-a0bf-4113-bacd-c735a955dd79" containerName="sequence-graph-ab362"
Jul 07 14:35:45.786853 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.786831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:45.790021 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.789994    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-fbc26-serving-cert\""
Jul 07 14:35:45.790354 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.790335    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-fbc26-kube-rbac-proxy-sar-config\""
Jul 07 14:35:45.799237 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.799211    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"]
Jul 07 14:35:45.830294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.830256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/441214a3-86a7-4c2c-b1d6-6aef7b931403-proxy-tls\") pod \"sequence-graph-fbc26-59cf79489b-jkc4g\" (UID: \"441214a3-86a7-4c2c-b1d6-6aef7b931403\") " pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:45.830294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.830294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/441214a3-86a7-4c2c-b1d6-6aef7b931403-openshift-service-ca-bundle\") pod \"sequence-graph-fbc26-59cf79489b-jkc4g\" (UID: \"441214a3-86a7-4c2c-b1d6-6aef7b931403\") " pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:45.931446 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.931397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/441214a3-86a7-4c2c-b1d6-6aef7b931403-proxy-tls\") pod \"sequence-graph-fbc26-59cf79489b-jkc4g\" (UID: \"441214a3-86a7-4c2c-b1d6-6aef7b931403\") " pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:45.931446 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.931437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/441214a3-86a7-4c2c-b1d6-6aef7b931403-openshift-service-ca-bundle\") pod \"sequence-graph-fbc26-59cf79489b-jkc4g\" (UID: \"441214a3-86a7-4c2c-b1d6-6aef7b931403\") " pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:45.931743 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:35:45.931558    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-fbc26-serving-cert: secret "sequence-graph-fbc26-serving-cert" not found
Jul 07 14:35:45.931743 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:35:45.931663    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/441214a3-86a7-4c2c-b1d6-6aef7b931403-proxy-tls podName:441214a3-86a7-4c2c-b1d6-6aef7b931403 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:46.431610615 +0000 UTC m=+1597.429201322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/441214a3-86a7-4c2c-b1d6-6aef7b931403-proxy-tls") pod "sequence-graph-fbc26-59cf79489b-jkc4g" (UID: "441214a3-86a7-4c2c-b1d6-6aef7b931403") : secret "sequence-graph-fbc26-serving-cert" not found
Jul 07 14:35:45.932170 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:45.932151    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/441214a3-86a7-4c2c-b1d6-6aef7b931403-openshift-service-ca-bundle\") pod \"sequence-graph-fbc26-59cf79489b-jkc4g\" (UID: \"441214a3-86a7-4c2c-b1d6-6aef7b931403\") " pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:46.434925 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:46.434887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/441214a3-86a7-4c2c-b1d6-6aef7b931403-proxy-tls\") pod \"sequence-graph-fbc26-59cf79489b-jkc4g\" (UID: \"441214a3-86a7-4c2c-b1d6-6aef7b931403\") " pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:46.437417 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:46.437390    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/441214a3-86a7-4c2c-b1d6-6aef7b931403-proxy-tls\") pod \"sequence-graph-fbc26-59cf79489b-jkc4g\" (UID: \"441214a3-86a7-4c2c-b1d6-6aef7b931403\") " pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:46.697222 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:46.697132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:46.828250 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:46.828212    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"]
Jul 07 14:35:46.831231 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:35:46.831202    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441214a3_86a7_4c2c_b1d6_6aef7b931403.slice/crio-39f27e48d6751c036679624fcd3d5579aa6e99d3ef4acb675aae8a840fc232ad WatchSource:0}: Error finding container 39f27e48d6751c036679624fcd3d5579aa6e99d3ef4acb675aae8a840fc232ad: Status 404 returned error can't find the container with id 39f27e48d6751c036679624fcd3d5579aa6e99d3ef4acb675aae8a840fc232ad
Jul 07 14:35:47.296568 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:47.296526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" event={"ID":"441214a3-86a7-4c2c-b1d6-6aef7b931403","Type":"ContainerStarted","Data":"54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e"}
Jul 07 14:35:47.296805 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:47.296585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" event={"ID":"441214a3-86a7-4c2c-b1d6-6aef7b931403","Type":"ContainerStarted","Data":"39f27e48d6751c036679624fcd3d5579aa6e99d3ef4acb675aae8a840fc232ad"}
Jul 07 14:35:47.296805 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:47.296663    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:47.315202 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:47.315142    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" podStartSLOduration=2.3151244 podStartE2EDuration="2.3151244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:47.313367235 +0000 UTC m=+1598.310957947" watchObservedRunningTime="2026-07-07 14:35:47.3151244 +0000 UTC m=+1598.312715111"
Jul 07 14:35:53.307512 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:53.307482    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:35:55.146221 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:55.146190    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:35:55.866512 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:55.866471    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"]
Jul 07 14:35:55.866744 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:55.866706    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26" containerID="cri-o://54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e" gracePeriod=30
Jul 07 14:35:56.018345 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.018314    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"]
Jul 07 14:35:56.018578 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.018542    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container" containerID="cri-o://16d93bba4583a2046528290364404e2e190709555d969da6a13d0139351afa93" gracePeriod=30
Jul 07 14:35:56.018693 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.018644    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kube-rbac-proxy" containerID="cri-o://9af9146840712ee315b302afd0039cf39bee54a834de8e5c0bb3ecbfb74fa60f" gracePeriod=30
Jul 07 14:35:56.020969 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.020943    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"]
Jul 07 14:35:56.024643 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.024604    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.027060 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.027040    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f31ce-predictor-serving-cert\""
Jul 07 14:35:56.027332 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.027318    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f31ce-kube-rbac-proxy-sar-config\""
Jul 07 14:35:56.037266 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.037242    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"]
Jul 07 14:35:56.110011 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.109963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.110166 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.110014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6f46bc9-feae-4385-8cda-0f73e34675fa-proxy-tls\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.110166 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.110098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4ct\" (UniqueName: \"kubernetes.io/projected/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-api-access-wn4ct\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.211240 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.211198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.211240 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.211243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6f46bc9-feae-4385-8cda-0f73e34675fa-proxy-tls\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.211881 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.211267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4ct\" (UniqueName: \"kubernetes.io/projected/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-api-access-wn4ct\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.211881 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:35:56.211361    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-serving-cert: secret "success-200-isvc-f31ce-predictor-serving-cert" not found
Jul 07 14:35:56.211881 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:35:56.211437    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6f46bc9-feae-4385-8cda-0f73e34675fa-proxy-tls podName:f6f46bc9-feae-4385-8cda-0f73e34675fa nodeName:}" failed. No retries permitted until 2026-07-07 14:35:56.711412001 +0000 UTC m=+1607.709002713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f6f46bc9-feae-4385-8cda-0f73e34675fa-proxy-tls") pod "success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" (UID: "f6f46bc9-feae-4385-8cda-0f73e34675fa") : secret "success-200-isvc-f31ce-predictor-serving-cert" not found
Jul 07 14:35:56.212154 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.212083    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.220022 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.220000    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4ct\" (UniqueName: \"kubernetes.io/projected/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-api-access-wn4ct\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.331577 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.331546    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerID="9af9146840712ee315b302afd0039cf39bee54a834de8e5c0bb3ecbfb74fa60f" exitCode=2
Jul 07 14:35:56.331788 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.331644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" event={"ID":"fd4bd77b-9822-4829-8e54-d6e24b6f264c","Type":"ContainerDied","Data":"9af9146840712ee315b302afd0039cf39bee54a834de8e5c0bb3ecbfb74fa60f"}
Jul 07 14:35:56.715700 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.715652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6f46bc9-feae-4385-8cda-0f73e34675fa-proxy-tls\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.718110 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.718084    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6f46bc9-feae-4385-8cda-0f73e34675fa-proxy-tls\") pod \"success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") " pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:56.941377 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:56.941336    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:57.075319 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:57.075145    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"]
Jul 07 14:35:57.078226 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:35:57.078195    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f46bc9_feae_4385_8cda_0f73e34675fa.slice/crio-756a8a582b975069a7f12f2abd97df5eab7e441668a78aa8729b94e5ae76fa32 WatchSource:0}: Error finding container 756a8a582b975069a7f12f2abd97df5eab7e441668a78aa8729b94e5ae76fa32: Status 404 returned error can't find the container with id 756a8a582b975069a7f12f2abd97df5eab7e441668a78aa8729b94e5ae76fa32
Jul 07 14:35:57.337312 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:57.337221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" event={"ID":"f6f46bc9-feae-4385-8cda-0f73e34675fa","Type":"ContainerStarted","Data":"e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5"}
Jul 07 14:35:57.337312 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:57.337286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" event={"ID":"f6f46bc9-feae-4385-8cda-0f73e34675fa","Type":"ContainerStarted","Data":"d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9"}
Jul 07 14:35:57.337312 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:57.337315    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" event={"ID":"f6f46bc9-feae-4385-8cda-0f73e34675fa","Type":"ContainerStarted","Data":"756a8a582b975069a7f12f2abd97df5eab7e441668a78aa8729b94e5ae76fa32"}
Jul 07 14:35:57.337814 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:57.337340    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:57.337814 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:57.337369    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:35:57.338955 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:57.338928    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:35:57.359073 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:57.359020    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podStartSLOduration=1.359002543 podStartE2EDuration="1.359002543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:57.358286409 +0000 UTC m=+1608.355877123" watchObservedRunningTime="2026-07-07 14:35:57.359002543 +0000 UTC m=+1608.356593260"
Jul 07 14:35:58.305777 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:58.305739    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:35:58.340843 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:58.340804    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:35:59.048489 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.048443    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 07 14:35:59.346558 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.346478    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerID="16d93bba4583a2046528290364404e2e190709555d969da6a13d0139351afa93" exitCode=0
Jul 07 14:35:59.346918 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.346558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" event={"ID":"fd4bd77b-9822-4829-8e54-d6e24b6f264c","Type":"ContainerDied","Data":"16d93bba4583a2046528290364404e2e190709555d969da6a13d0139351afa93"}
Jul 07 14:35:59.375793 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.375767    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:35:59.439527 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.439490    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n9cc\" (UniqueName: \"kubernetes.io/projected/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-api-access-4n9cc\") pod \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") "
Jul 07 14:35:59.439726 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.439561    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd4bd77b-9822-4829-8e54-d6e24b6f264c-proxy-tls\") pod \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") "
Jul 07 14:35:59.439726 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.439586    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-rbac-proxy-sar-config\") pod \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\" (UID: \"fd4bd77b-9822-4829-8e54-d6e24b6f264c\") "
Jul 07 14:35:59.439981 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.439959    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd4bd77b-9822-4829-8e54-d6e24b6f264c" (UID: "fd4bd77b-9822-4829-8e54-d6e24b6f264c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:35:59.441656 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.441639    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-api-access-4n9cc" (OuterVolumeSpecName: "kube-api-access-4n9cc") pod "fd4bd77b-9822-4829-8e54-d6e24b6f264c" (UID: "fd4bd77b-9822-4829-8e54-d6e24b6f264c"). InnerVolumeSpecName "kube-api-access-4n9cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:35:59.441774 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.441751    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd4bd77b-9822-4829-8e54-d6e24b6f264c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd4bd77b-9822-4829-8e54-d6e24b6f264c" (UID: "fd4bd77b-9822-4829-8e54-d6e24b6f264c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:35:59.540761 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.540726    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n9cc\" (UniqueName: \"kubernetes.io/projected/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-api-access-4n9cc\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:35:59.540761 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.540757    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd4bd77b-9822-4829-8e54-d6e24b6f264c-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:35:59.540761 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:35:59.540767    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd4bd77b-9822-4829-8e54-d6e24b6f264c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:36:00.351919 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:00.351881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv" event={"ID":"fd4bd77b-9822-4829-8e54-d6e24b6f264c","Type":"ContainerDied","Data":"8d2ed63fbb3ac57f73f6916df116325d5d8cac173b44d1c095e9fecb28a0f906"}
Jul 07 14:36:00.351919 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:00.351904    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"
Jul 07 14:36:00.352458 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:00.351974    2565 scope.go:117] "RemoveContainer" containerID="9af9146840712ee315b302afd0039cf39bee54a834de8e5c0bb3ecbfb74fa60f"
Jul 07 14:36:00.359807 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:00.359784    2565 scope.go:117] "RemoveContainer" containerID="16d93bba4583a2046528290364404e2e190709555d969da6a13d0139351afa93"
Jul 07 14:36:00.370755 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:00.370729    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"]
Jul 07 14:36:00.374705 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:00.374677    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc26-predictor-b5c7dd9f9-gkdxv"]
Jul 07 14:36:01.485981 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:01.485884    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" path="/var/lib/kubelet/pods/fd4bd77b-9822-4829-8e54-d6e24b6f264c/volumes"
Jul 07 14:36:03.305234 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:03.305187    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:36:03.346288 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:03.346259    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:36:03.346845 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:03.346816    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:36:06.383839 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.383800    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"]
Jul 07 14:36:06.384297 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.384268    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kube-rbac-proxy"
Jul 07 14:36:06.384297 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.384286    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kube-rbac-proxy"
Jul 07 14:36:06.384413 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.384322    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container"
Jul 07 14:36:06.384413 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.384331    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container"
Jul 07 14:36:06.384413 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.384404    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kube-rbac-proxy"
Jul 07 14:36:06.384547 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.384416    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd4bd77b-9822-4829-8e54-d6e24b6f264c" containerName="kserve-container"
Jul 07 14:36:06.388727 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.388706    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:06.390985 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.390965    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fe15b-serving-cert\""
Jul 07 14:36:06.390985 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.390974    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fe15b-kube-rbac-proxy-sar-config\""
Jul 07 14:36:06.440280 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.440248    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"]
Jul 07 14:36:06.500054 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.500011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10971237-a1b1-46e5-bbd2-24d949011576-openshift-service-ca-bundle\") pod \"ensemble-graph-fe15b-7ffc6fdfb6-xk2pm\" (UID: \"10971237-a1b1-46e5-bbd2-24d949011576\") " pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:06.500230 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.500067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10971237-a1b1-46e5-bbd2-24d949011576-proxy-tls\") pod \"ensemble-graph-fe15b-7ffc6fdfb6-xk2pm\" (UID: \"10971237-a1b1-46e5-bbd2-24d949011576\") " pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:06.600490 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.600445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10971237-a1b1-46e5-bbd2-24d949011576-proxy-tls\") pod \"ensemble-graph-fe15b-7ffc6fdfb6-xk2pm\" (UID: \"10971237-a1b1-46e5-bbd2-24d949011576\") " pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:06.600696 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.600538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10971237-a1b1-46e5-bbd2-24d949011576-openshift-service-ca-bundle\") pod \"ensemble-graph-fe15b-7ffc6fdfb6-xk2pm\" (UID: \"10971237-a1b1-46e5-bbd2-24d949011576\") " pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:06.600696 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:36:06.600599    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-fe15b-serving-cert: secret "ensemble-graph-fe15b-serving-cert" not found
Jul 07 14:36:06.600696 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:36:06.600688    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10971237-a1b1-46e5-bbd2-24d949011576-proxy-tls podName:10971237-a1b1-46e5-bbd2-24d949011576 nodeName:}" failed. No retries permitted until 2026-07-07 14:36:07.100670619 +0000 UTC m=+1618.098261312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/10971237-a1b1-46e5-bbd2-24d949011576-proxy-tls") pod "ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" (UID: "10971237-a1b1-46e5-bbd2-24d949011576") : secret "ensemble-graph-fe15b-serving-cert" not found
Jul 07 14:36:06.601212 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:06.601187    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10971237-a1b1-46e5-bbd2-24d949011576-openshift-service-ca-bundle\") pod \"ensemble-graph-fe15b-7ffc6fdfb6-xk2pm\" (UID: \"10971237-a1b1-46e5-bbd2-24d949011576\") " pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:07.105922 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:07.105877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10971237-a1b1-46e5-bbd2-24d949011576-proxy-tls\") pod \"ensemble-graph-fe15b-7ffc6fdfb6-xk2pm\" (UID: \"10971237-a1b1-46e5-bbd2-24d949011576\") " pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:07.108383 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:07.108356    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10971237-a1b1-46e5-bbd2-24d949011576-proxy-tls\") pod \"ensemble-graph-fe15b-7ffc6fdfb6-xk2pm\" (UID: \"10971237-a1b1-46e5-bbd2-24d949011576\") " pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:07.298827 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:07.298787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:07.438547 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:07.438521    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"]
Jul 07 14:36:07.441129 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:36:07.441101    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10971237_a1b1_46e5_bbd2_24d949011576.slice/crio-863f4fbac4cfdab605849f5a9756d1798fb4180aa8f30728e3ced08a90355157 WatchSource:0}: Error finding container 863f4fbac4cfdab605849f5a9756d1798fb4180aa8f30728e3ced08a90355157: Status 404 returned error can't find the container with id 863f4fbac4cfdab605849f5a9756d1798fb4180aa8f30728e3ced08a90355157
Jul 07 14:36:08.305477 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:08.305432    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:36:08.305676 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:08.305538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:36:08.385555 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:08.385520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" event={"ID":"10971237-a1b1-46e5-bbd2-24d949011576","Type":"ContainerStarted","Data":"2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac"}
Jul 07 14:36:08.385718 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:08.385569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" event={"ID":"10971237-a1b1-46e5-bbd2-24d949011576","Type":"ContainerStarted","Data":"863f4fbac4cfdab605849f5a9756d1798fb4180aa8f30728e3ced08a90355157"}
Jul 07 14:36:08.385718 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:08.385656    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:08.407132 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:08.407080    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" podStartSLOduration=2.407064774 podStartE2EDuration="2.407064774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:36:08.406514552 +0000 UTC m=+1619.404105269" watchObservedRunningTime="2026-07-07 14:36:08.407064774 +0000 UTC m=+1619.404655488"
Jul 07 14:36:13.305472 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:13.305434    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:36:13.346765 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:13.346725    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:36:14.394648 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:14.394602    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:36:18.305093 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:18.305050    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:36:23.305360 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:23.305308    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:36:23.347079 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:23.347038    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:36:26.014274 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.014245    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:36:26.168138 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.168049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/441214a3-86a7-4c2c-b1d6-6aef7b931403-openshift-service-ca-bundle\") pod \"441214a3-86a7-4c2c-b1d6-6aef7b931403\" (UID: \"441214a3-86a7-4c2c-b1d6-6aef7b931403\") "
Jul 07 14:36:26.168138 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.168102    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/441214a3-86a7-4c2c-b1d6-6aef7b931403-proxy-tls\") pod \"441214a3-86a7-4c2c-b1d6-6aef7b931403\" (UID: \"441214a3-86a7-4c2c-b1d6-6aef7b931403\") "
Jul 07 14:36:26.168449 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.168427    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441214a3-86a7-4c2c-b1d6-6aef7b931403-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "441214a3-86a7-4c2c-b1d6-6aef7b931403" (UID: "441214a3-86a7-4c2c-b1d6-6aef7b931403"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:36:26.170245 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.170223    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/441214a3-86a7-4c2c-b1d6-6aef7b931403-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "441214a3-86a7-4c2c-b1d6-6aef7b931403" (UID: "441214a3-86a7-4c2c-b1d6-6aef7b931403"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:36:26.269410 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.269363    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/441214a3-86a7-4c2c-b1d6-6aef7b931403-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:36:26.269410 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.269402    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/441214a3-86a7-4c2c-b1d6-6aef7b931403-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:36:26.456670 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.456562    2565 generic.go:356] "Generic (PLEG): container finished" podID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerID="54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e" exitCode=0
Jul 07 14:36:26.456670 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.456610    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"
Jul 07 14:36:26.456670 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.456650    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" event={"ID":"441214a3-86a7-4c2c-b1d6-6aef7b931403","Type":"ContainerDied","Data":"54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e"}
Jul 07 14:36:26.456883 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.456709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g" event={"ID":"441214a3-86a7-4c2c-b1d6-6aef7b931403","Type":"ContainerDied","Data":"39f27e48d6751c036679624fcd3d5579aa6e99d3ef4acb675aae8a840fc232ad"}
Jul 07 14:36:26.456883 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.456732    2565 scope.go:117] "RemoveContainer" containerID="54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e"
Jul 07 14:36:26.465514 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.465495    2565 scope.go:117] "RemoveContainer" containerID="54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e"
Jul 07 14:36:26.465870 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:36:26.465851    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e\": container with ID starting with 54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e not found: ID does not exist" containerID="54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e"
Jul 07 14:36:26.465938 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.465879    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e"} err="failed to get container status \"54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e\": rpc error: code = NotFound desc = could not find container \"54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e\": container with ID starting with 54d847bc67c1e52bd9fc84b9b11cafcc439f921a81541ea0551f1952d574668e not found: ID does not exist"
Jul 07 14:36:26.478491 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.478461    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"]
Jul 07 14:36:26.487863 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:26.487826    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-fbc26-59cf79489b-jkc4g"]
Jul 07 14:36:27.486068 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:27.486036    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" path="/var/lib/kubelet/pods/441214a3-86a7-4c2c-b1d6-6aef7b931403/volumes"
Jul 07 14:36:33.346990 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:33.346952    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:36:43.347852 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:43.347815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:36:56.120784 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.120743    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"]
Jul 07 14:36:56.121255 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.121207    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26"
Jul 07 14:36:56.121255 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.121227    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26"
Jul 07 14:36:56.121365 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.121298    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="441214a3-86a7-4c2c-b1d6-6aef7b931403" containerName="sequence-graph-fbc26"
Jul 07 14:36:56.124292 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.124270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:36:56.127217 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.127190    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f31ce-kube-rbac-proxy-sar-config\""
Jul 07 14:36:56.127482 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.127464    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f31ce-serving-cert\""
Jul 07 14:36:56.135372 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.135346    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"]
Jul 07 14:36:56.201964 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.201931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b2a024a-5068-40c0-8124-bc69b169e130-openshift-service-ca-bundle\") pod \"sequence-graph-f31ce-9c588fcbb-qjxf8\" (UID: \"4b2a024a-5068-40c0-8124-bc69b169e130\") " pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:36:56.201964 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.201978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b2a024a-5068-40c0-8124-bc69b169e130-proxy-tls\") pod \"sequence-graph-f31ce-9c588fcbb-qjxf8\" (UID: \"4b2a024a-5068-40c0-8124-bc69b169e130\") " pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:36:56.303384 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.303339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b2a024a-5068-40c0-8124-bc69b169e130-openshift-service-ca-bundle\") pod \"sequence-graph-f31ce-9c588fcbb-qjxf8\" (UID: \"4b2a024a-5068-40c0-8124-bc69b169e130\") " pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:36:56.303384 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.303386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b2a024a-5068-40c0-8124-bc69b169e130-proxy-tls\") pod \"sequence-graph-f31ce-9c588fcbb-qjxf8\" (UID: \"4b2a024a-5068-40c0-8124-bc69b169e130\") " pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:36:56.303659 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:36:56.303505    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-f31ce-serving-cert: secret "sequence-graph-f31ce-serving-cert" not found
Jul 07 14:36:56.303659 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:36:56.303575    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2a024a-5068-40c0-8124-bc69b169e130-proxy-tls podName:4b2a024a-5068-40c0-8124-bc69b169e130 nodeName:}" failed. No retries permitted until 2026-07-07 14:36:56.803556274 +0000 UTC m=+1667.801146967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4b2a024a-5068-40c0-8124-bc69b169e130-proxy-tls") pod "sequence-graph-f31ce-9c588fcbb-qjxf8" (UID: "4b2a024a-5068-40c0-8124-bc69b169e130") : secret "sequence-graph-f31ce-serving-cert" not found
Jul 07 14:36:56.304128 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.304108    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b2a024a-5068-40c0-8124-bc69b169e130-openshift-service-ca-bundle\") pod \"sequence-graph-f31ce-9c588fcbb-qjxf8\" (UID: \"4b2a024a-5068-40c0-8124-bc69b169e130\") " pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:36:56.807939 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.807898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b2a024a-5068-40c0-8124-bc69b169e130-proxy-tls\") pod \"sequence-graph-f31ce-9c588fcbb-qjxf8\" (UID: \"4b2a024a-5068-40c0-8124-bc69b169e130\") " pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:36:56.810333 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:56.810306    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b2a024a-5068-40c0-8124-bc69b169e130-proxy-tls\") pod \"sequence-graph-f31ce-9c588fcbb-qjxf8\" (UID: \"4b2a024a-5068-40c0-8124-bc69b169e130\") " pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:36:57.034333 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:57.034302    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:36:57.178868 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:57.178784    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"]
Jul 07 14:36:57.181644 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:36:57.181599    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2a024a_5068_40c0_8124_bc69b169e130.slice/crio-22401f3e28d10ec0366d023f068e4bc8c3f1de1a57298f7622c8f9545545be79 WatchSource:0}: Error finding container 22401f3e28d10ec0366d023f068e4bc8c3f1de1a57298f7622c8f9545545be79: Status 404 returned error can't find the container with id 22401f3e28d10ec0366d023f068e4bc8c3f1de1a57298f7622c8f9545545be79
Jul 07 14:36:57.573396 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:57.573355    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" event={"ID":"4b2a024a-5068-40c0-8124-bc69b169e130","Type":"ContainerStarted","Data":"508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450"}
Jul 07 14:36:57.573570 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:57.573417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" event={"ID":"4b2a024a-5068-40c0-8124-bc69b169e130","Type":"ContainerStarted","Data":"22401f3e28d10ec0366d023f068e4bc8c3f1de1a57298f7622c8f9545545be79"}
Jul 07 14:36:57.573570 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:36:57.573505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:37:03.583611 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:37:03.583579    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:37:03.602968 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:37:03.602917    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" podStartSLOduration=7.602898207 podStartE2EDuration="7.602898207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:36:57.605141799 +0000 UTC m=+1668.602732514" watchObservedRunningTime="2026-07-07 14:37:03.602898207 +0000 UTC m=+1674.600488923"
Jul 07 14:44:21.092082 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.092040    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"]
Jul 07 14:44:21.094535 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.092336    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b" containerID="cri-o://2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac" gracePeriod=30
Jul 07 14:44:21.284543 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.284513    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"]
Jul 07 14:44:21.287883 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.287860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.290569 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.290530    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6e67f-predictor-serving-cert\""
Jul 07 14:44:21.290740 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.290611    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6e67f-kube-rbac-proxy-sar-config\""
Jul 07 14:44:21.296445 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.296424    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"]
Jul 07 14:44:21.296827 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.296644    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container" containerID="cri-o://7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2" gracePeriod=30
Jul 07 14:44:21.296827 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.296719    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kube-rbac-proxy" containerID="cri-o://bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435" gracePeriod=30
Jul 07 14:44:21.298875 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.298852    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"]
Jul 07 14:44:21.362092 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.361991    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.362576 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.362534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f271e4c-1439-419a-8b46-04b1c7b34595-proxy-tls\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.362730 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.362642    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzlh\" (UniqueName: \"kubernetes.io/projected/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-api-access-nlzlh\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.463842 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.463794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.464050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.463869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f271e4c-1439-419a-8b46-04b1c7b34595-proxy-tls\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.464050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.463915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzlh\" (UniqueName: \"kubernetes.io/projected/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-api-access-nlzlh\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.464291 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:44:21.464266    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-serving-cert: secret "success-200-isvc-6e67f-predictor-serving-cert" not found
Jul 07 14:44:21.464363 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:44:21.464351    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f271e4c-1439-419a-8b46-04b1c7b34595-proxy-tls podName:7f271e4c-1439-419a-8b46-04b1c7b34595 nodeName:}" failed. No retries permitted until 2026-07-07 14:44:21.964331581 +0000 UTC m=+2112.961922274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f271e4c-1439-419a-8b46-04b1c7b34595-proxy-tls") pod "success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" (UID: "7f271e4c-1439-419a-8b46-04b1c7b34595") : secret "success-200-isvc-6e67f-predictor-serving-cert" not found
Jul 07 14:44:21.464577 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.464555    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.474167 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.474140    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzlh\" (UniqueName: \"kubernetes.io/projected/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-api-access-nlzlh\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.968237 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.968201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f271e4c-1439-419a-8b46-04b1c7b34595-proxy-tls\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:21.970692 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:21.970666    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f271e4c-1439-419a-8b46-04b1c7b34595-proxy-tls\") pod \"success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") " pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:22.200313 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:22.200264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:22.230456 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:22.230366    2565 generic.go:356] "Generic (PLEG): container finished" podID="963cf95d-8dda-4225-b8e7-d945024f7253" containerID="bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435" exitCode=2
Jul 07 14:44:22.230456 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:22.230423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" event={"ID":"963cf95d-8dda-4225-b8e7-d945024f7253","Type":"ContainerDied","Data":"bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435"}
Jul 07 14:44:22.335892 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:22.335858    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"]
Jul 07 14:44:22.338885 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:44:22.338852    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f271e4c_1439_419a_8b46_04b1c7b34595.slice/crio-9df1dd4d76138695cb2e2f2e23ad2328e0d352c6d486b1a658058bbbb4569bc0 WatchSource:0}: Error finding container 9df1dd4d76138695cb2e2f2e23ad2328e0d352c6d486b1a658058bbbb4569bc0: Status 404 returned error can't find the container with id 9df1dd4d76138695cb2e2f2e23ad2328e0d352c6d486b1a658058bbbb4569bc0
Jul 07 14:44:23.236683 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:23.236609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" event={"ID":"7f271e4c-1439-419a-8b46-04b1c7b34595","Type":"ContainerStarted","Data":"b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152"}
Jul 07 14:44:23.237172 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:23.236714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" event={"ID":"7f271e4c-1439-419a-8b46-04b1c7b34595","Type":"ContainerStarted","Data":"420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea"}
Jul 07 14:44:23.237172 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:23.236745    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" event={"ID":"7f271e4c-1439-419a-8b46-04b1c7b34595","Type":"ContainerStarted","Data":"9df1dd4d76138695cb2e2f2e23ad2328e0d352c6d486b1a658058bbbb4569bc0"}
Jul 07 14:44:23.237172 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:23.236865    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:23.260434 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:23.260359    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" podStartSLOduration=2.260339706 podStartE2EDuration="2.260339706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:23.259159997 +0000 UTC m=+2114.256750723" watchObservedRunningTime="2026-07-07 14:44:23.260339706 +0000 UTC m=+2114.257930422"
Jul 07 14:44:24.240713 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.240670    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:24.241987 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.241954    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 14:44:24.393582 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.393550    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:44:24.550519 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.550488    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:44:24.590966 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.590925    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963cf95d-8dda-4225-b8e7-d945024f7253-proxy-tls\") pod \"963cf95d-8dda-4225-b8e7-d945024f7253\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") "
Jul 07 14:44:24.590966 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.590976    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbtb8\" (UniqueName: \"kubernetes.io/projected/963cf95d-8dda-4225-b8e7-d945024f7253-kube-api-access-kbtb8\") pod \"963cf95d-8dda-4225-b8e7-d945024f7253\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") "
Jul 07 14:44:24.591232 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.591039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963cf95d-8dda-4225-b8e7-d945024f7253-kube-rbac-proxy-sar-config\") pod \"963cf95d-8dda-4225-b8e7-d945024f7253\" (UID: \"963cf95d-8dda-4225-b8e7-d945024f7253\") "
Jul 07 14:44:24.591521 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.591484    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963cf95d-8dda-4225-b8e7-d945024f7253-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "963cf95d-8dda-4225-b8e7-d945024f7253" (UID: "963cf95d-8dda-4225-b8e7-d945024f7253"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:44:24.593170 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.593149    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963cf95d-8dda-4225-b8e7-d945024f7253-kube-api-access-kbtb8" (OuterVolumeSpecName: "kube-api-access-kbtb8") pod "963cf95d-8dda-4225-b8e7-d945024f7253" (UID: "963cf95d-8dda-4225-b8e7-d945024f7253"). InnerVolumeSpecName "kube-api-access-kbtb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:44:24.593247 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.593153    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963cf95d-8dda-4225-b8e7-d945024f7253-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "963cf95d-8dda-4225-b8e7-d945024f7253" (UID: "963cf95d-8dda-4225-b8e7-d945024f7253"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:44:24.691673 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.691591    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/963cf95d-8dda-4225-b8e7-d945024f7253-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:44:24.691673 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.691671    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/963cf95d-8dda-4225-b8e7-d945024f7253-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:44:24.691888 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:24.691691    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbtb8\" (UniqueName: \"kubernetes.io/projected/963cf95d-8dda-4225-b8e7-d945024f7253-kube-api-access-kbtb8\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:44:25.246101 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.246066    2565 generic.go:356] "Generic (PLEG): container finished" podID="963cf95d-8dda-4225-b8e7-d945024f7253" containerID="7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2" exitCode=0
Jul 07 14:44:25.246508 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.246124    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"
Jul 07 14:44:25.246508 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.246148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" event={"ID":"963cf95d-8dda-4225-b8e7-d945024f7253","Type":"ContainerDied","Data":"7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2"}
Jul 07 14:44:25.246508 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.246204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4" event={"ID":"963cf95d-8dda-4225-b8e7-d945024f7253","Type":"ContainerDied","Data":"c75bae53f698dfb11a8cb3a82fd1e5dcb44bfc72a82797f0eeae55ade87e3778"}
Jul 07 14:44:25.246508 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.246229    2565 scope.go:117] "RemoveContainer" containerID="bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435"
Jul 07 14:44:25.246854 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.246829    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 14:44:25.254941 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.254923    2565 scope.go:117] "RemoveContainer" containerID="7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2"
Jul 07 14:44:25.262582 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.262562    2565 scope.go:117] "RemoveContainer" containerID="bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435"
Jul 07 14:44:25.262858 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:44:25.262840    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435\": container with ID starting with bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435 not found: ID does not exist" containerID="bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435"
Jul 07 14:44:25.262909 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.262869    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435"} err="failed to get container status \"bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435\": rpc error: code = NotFound desc = could not find container \"bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435\": container with ID starting with bea6b637d0029393bfa35a41fbc63457a248e99459ef5f7f49e3b49c8bdb1435 not found: ID does not exist"
Jul 07 14:44:25.262909 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.262886    2565 scope.go:117] "RemoveContainer" containerID="7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2"
Jul 07 14:44:25.263101 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:44:25.263080    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2\": container with ID starting with 7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2 not found: ID does not exist" containerID="7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2"
Jul 07 14:44:25.263154 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.263110    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2"} err="failed to get container status \"7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2\": rpc error: code = NotFound desc = could not find container \"7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2\": container with ID starting with 7373153a693300bc00fa38bee79e3e68f699d0b564c2b0841b5b1df0aed9f3f2 not found: ID does not exist"
Jul 07 14:44:25.267708 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.267685    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"]
Jul 07 14:44:25.273202 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.273178    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe15b-predictor-6654f94d55-dmfx4"]
Jul 07 14:44:25.487414 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:25.487382    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" path="/var/lib/kubelet/pods/963cf95d-8dda-4225-b8e7-d945024f7253/volumes"
Jul 07 14:44:29.393070 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:29.393034    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:44:30.251467 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:30.251430    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:44:30.252037 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:30.252010    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 14:44:34.393160 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:34.393120    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:44:34.393665 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:34.393247    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:44:39.393436 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:39.393389    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:44:40.252343 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:40.252296    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 14:44:44.393405 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:44.393365    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:44:49.393826 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:49.393785    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:44:50.252909 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:50.252864    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 14:44:51.239317 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.239290    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:44:51.310670 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.310613    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10971237-a1b1-46e5-bbd2-24d949011576-proxy-tls\") pod \"10971237-a1b1-46e5-bbd2-24d949011576\" (UID: \"10971237-a1b1-46e5-bbd2-24d949011576\") "
Jul 07 14:44:51.310841 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.310697    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10971237-a1b1-46e5-bbd2-24d949011576-openshift-service-ca-bundle\") pod \"10971237-a1b1-46e5-bbd2-24d949011576\" (UID: \"10971237-a1b1-46e5-bbd2-24d949011576\") "
Jul 07 14:44:51.311063 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.311040    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10971237-a1b1-46e5-bbd2-24d949011576-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "10971237-a1b1-46e5-bbd2-24d949011576" (UID: "10971237-a1b1-46e5-bbd2-24d949011576"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:44:51.312724 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.312704    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10971237-a1b1-46e5-bbd2-24d949011576-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10971237-a1b1-46e5-bbd2-24d949011576" (UID: "10971237-a1b1-46e5-bbd2-24d949011576"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:44:51.352462 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.352378    2565 generic.go:356] "Generic (PLEG): container finished" podID="10971237-a1b1-46e5-bbd2-24d949011576" containerID="2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac" exitCode=0
Jul 07 14:44:51.352462 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.352432    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"
Jul 07 14:44:51.352462 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.352441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" event={"ID":"10971237-a1b1-46e5-bbd2-24d949011576","Type":"ContainerDied","Data":"2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac"}
Jul 07 14:44:51.352720 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.352493    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm" event={"ID":"10971237-a1b1-46e5-bbd2-24d949011576","Type":"ContainerDied","Data":"863f4fbac4cfdab605849f5a9756d1798fb4180aa8f30728e3ced08a90355157"}
Jul 07 14:44:51.352720 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.352526    2565 scope.go:117] "RemoveContainer" containerID="2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac"
Jul 07 14:44:51.360872 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.360854    2565 scope.go:117] "RemoveContainer" containerID="2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac"
Jul 07 14:44:51.361183 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:44:51.361161    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac\": container with ID starting with 2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac not found: ID does not exist" containerID="2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac"
Jul 07 14:44:51.361226 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.361192    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac"} err="failed to get container status \"2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac\": rpc error: code = NotFound desc = could not find container \"2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac\": container with ID starting with 2ab2e4311b826e5054a6f6c0c2e4b485aac13da1bf9da12c05f971490320c4ac not found: ID does not exist"
Jul 07 14:44:51.373094 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.373060    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"]
Jul 07 14:44:51.376806 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.376780    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fe15b-7ffc6fdfb6-xk2pm"]
Jul 07 14:44:51.411889 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.411857    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10971237-a1b1-46e5-bbd2-24d949011576-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:44:51.411889 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.411889    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10971237-a1b1-46e5-bbd2-24d949011576-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:44:51.486942 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:44:51.486909    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10971237-a1b1-46e5-bbd2-24d949011576" path="/var/lib/kubelet/pods/10971237-a1b1-46e5-bbd2-24d949011576/volumes"
Jul 07 14:45:00.252258 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:00.252214    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 07 14:45:10.253247 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.253212    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:45:10.839963 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.839921    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"]
Jul 07 14:45:10.840220 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.840194    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce" containerID="cri-o://508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450" gracePeriod=30
Jul 07 14:45:10.950294 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.950262    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"]
Jul 07 14:45:10.950523 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.950498    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container" containerID="cri-o://d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9" gracePeriod=30
Jul 07 14:45:10.950645 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.950576    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kube-rbac-proxy" containerID="cri-o://e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5" gracePeriod=30
Jul 07 14:45:10.974365 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974333    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"]
Jul 07 14:45:10.974656 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974643    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b"
Jul 07 14:45:10.974707 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974658    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b"
Jul 07 14:45:10.974707 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974669    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container"
Jul 07 14:45:10.974707 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974675    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container"
Jul 07 14:45:10.974707 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974689    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kube-rbac-proxy"
Jul 07 14:45:10.974707 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974695    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kube-rbac-proxy"
Jul 07 14:45:10.974855 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974749    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kserve-container"
Jul 07 14:45:10.974855 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974757    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="963cf95d-8dda-4225-b8e7-d945024f7253" containerName="kube-rbac-proxy"
Jul 07 14:45:10.974855 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.974764    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="10971237-a1b1-46e5-bbd2-24d949011576" containerName="ensemble-graph-fe15b"
Jul 07 14:45:10.979096 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.979074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:10.981363 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.981336    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-36f00-predictor-serving-cert\""
Jul 07 14:45:10.981477 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.981367    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-36f00-kube-rbac-proxy-sar-config\""
Jul 07 14:45:10.987308 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:10.987283    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"]
Jul 07 14:45:11.080234 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.080203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6x7n\" (UniqueName: \"kubernetes.io/projected/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-api-access-l6x7n\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.080388 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.080254    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c525a2c-686b-4c1c-afd0-e826d769d80a-proxy-tls\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.080388 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.080277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.181630 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.181566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c525a2c-686b-4c1c-afd0-e826d769d80a-proxy-tls\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.181823 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.181649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.181823 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:11.181720    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-36f00-predictor-serving-cert: secret "success-200-isvc-36f00-predictor-serving-cert" not found
Jul 07 14:45:11.181823 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.181769    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6x7n\" (UniqueName: \"kubernetes.io/projected/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-api-access-l6x7n\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.181823 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:11.181799    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c525a2c-686b-4c1c-afd0-e826d769d80a-proxy-tls podName:3c525a2c-686b-4c1c-afd0-e826d769d80a nodeName:}" failed. No retries permitted until 2026-07-07 14:45:11.681777421 +0000 UTC m=+2162.679368116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3c525a2c-686b-4c1c-afd0-e826d769d80a-proxy-tls") pod "success-200-isvc-36f00-predictor-869b59995f-knmkk" (UID: "3c525a2c-686b-4c1c-afd0-e826d769d80a") : secret "success-200-isvc-36f00-predictor-serving-cert" not found
Jul 07 14:45:11.182312 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.182294    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.196175 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.196143    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6x7n\" (UniqueName: \"kubernetes.io/projected/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-api-access-l6x7n\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.429595 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.429555    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerID="e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5" exitCode=2
Jul 07 14:45:11.430005 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.429655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" event={"ID":"f6f46bc9-feae-4385-8cda-0f73e34675fa","Type":"ContainerDied","Data":"e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5"}
Jul 07 14:45:11.685884 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.685832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c525a2c-686b-4c1c-afd0-e826d769d80a-proxy-tls\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.688255 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.688225    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c525a2c-686b-4c1c-afd0-e826d769d80a-proxy-tls\") pod \"success-200-isvc-36f00-predictor-869b59995f-knmkk\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") " pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:11.890495 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:11.890454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:12.024007 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:12.023985    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"]
Jul 07 14:45:12.026507 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:45:12.026481    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c525a2c_686b_4c1c_afd0_e826d769d80a.slice/crio-858206f349ad5ca0539a48faf6d283937db30f340d69a65085cff2feab05a5e4 WatchSource:0}: Error finding container 858206f349ad5ca0539a48faf6d283937db30f340d69a65085cff2feab05a5e4: Status 404 returned error can't find the container with id 858206f349ad5ca0539a48faf6d283937db30f340d69a65085cff2feab05a5e4
Jul 07 14:45:12.436949 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:12.436913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" event={"ID":"3c525a2c-686b-4c1c-afd0-e826d769d80a","Type":"ContainerStarted","Data":"f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e"}
Jul 07 14:45:12.437363 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:12.436978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" event={"ID":"3c525a2c-686b-4c1c-afd0-e826d769d80a","Type":"ContainerStarted","Data":"cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656"}
Jul 07 14:45:12.437363 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:12.437009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" event={"ID":"3c525a2c-686b-4c1c-afd0-e826d769d80a","Type":"ContainerStarted","Data":"858206f349ad5ca0539a48faf6d283937db30f340d69a65085cff2feab05a5e4"}
Jul 07 14:45:12.437363 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:12.437120    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:12.454936 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:12.454875    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" podStartSLOduration=2.454845673 podStartE2EDuration="2.454845673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:45:12.454591374 +0000 UTC m=+2163.452182114" watchObservedRunningTime="2026-07-07 14:45:12.454845673 +0000 UTC m=+2163.452436386"
Jul 07 14:45:13.341692 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:13.341646    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 07 14:45:13.346749 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:13.346713    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 14:45:13.440316 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:13.440279    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:13.441677 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:13.441651    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:45:13.582646 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:13.582586    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:14.307168 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.307142    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:45:14.407789 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.407741    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6f46bc9-feae-4385-8cda-0f73e34675fa-proxy-tls\") pod \"f6f46bc9-feae-4385-8cda-0f73e34675fa\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") "
Jul 07 14:45:14.407789 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.407796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-rbac-proxy-sar-config\") pod \"f6f46bc9-feae-4385-8cda-0f73e34675fa\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") "
Jul 07 14:45:14.408004 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.407877    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn4ct\" (UniqueName: \"kubernetes.io/projected/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-api-access-wn4ct\") pod \"f6f46bc9-feae-4385-8cda-0f73e34675fa\" (UID: \"f6f46bc9-feae-4385-8cda-0f73e34675fa\") "
Jul 07 14:45:14.408220 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.408195    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f6f46bc9-feae-4385-8cda-0f73e34675fa" (UID: "f6f46bc9-feae-4385-8cda-0f73e34675fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:45:14.409924 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.409900    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-api-access-wn4ct" (OuterVolumeSpecName: "kube-api-access-wn4ct") pod "f6f46bc9-feae-4385-8cda-0f73e34675fa" (UID: "f6f46bc9-feae-4385-8cda-0f73e34675fa"). InnerVolumeSpecName "kube-api-access-wn4ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:45:14.410045 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.410023    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f46bc9-feae-4385-8cda-0f73e34675fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f6f46bc9-feae-4385-8cda-0f73e34675fa" (UID: "f6f46bc9-feae-4385-8cda-0f73e34675fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:45:14.444857 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.444821    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerID="d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9" exitCode=0
Jul 07 14:45:14.445327 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.444876    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"
Jul 07 14:45:14.445327 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.444887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" event={"ID":"f6f46bc9-feae-4385-8cda-0f73e34675fa","Type":"ContainerDied","Data":"d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9"}
Jul 07 14:45:14.445327 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.444963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb" event={"ID":"f6f46bc9-feae-4385-8cda-0f73e34675fa","Type":"ContainerDied","Data":"756a8a582b975069a7f12f2abd97df5eab7e441668a78aa8729b94e5ae76fa32"}
Jul 07 14:45:14.445327 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.445006    2565 scope.go:117] "RemoveContainer" containerID="e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5"
Jul 07 14:45:14.445559 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.445536    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:45:14.458764 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.458744    2565 scope.go:117] "RemoveContainer" containerID="d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9"
Jul 07 14:45:14.466238 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.466219    2565 scope.go:117] "RemoveContainer" containerID="e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5"
Jul 07 14:45:14.466508 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:14.466486    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5\": container with ID starting with e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5 not found: ID does not exist" containerID="e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5"
Jul 07 14:45:14.466559 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.466518    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5"} err="failed to get container status \"e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5\": rpc error: code = NotFound desc = could not find container \"e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5\": container with ID starting with e974c766eeabc2c3fe201ae9b23e447203316325d2d5c6140a6923de0391bda5 not found: ID does not exist"
Jul 07 14:45:14.466559 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.466536    2565 scope.go:117] "RemoveContainer" containerID="d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9"
Jul 07 14:45:14.466801 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:14.466773    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9\": container with ID starting with d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9 not found: ID does not exist" containerID="d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9"
Jul 07 14:45:14.466856 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.466813    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9"} err="failed to get container status \"d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9\": rpc error: code = NotFound desc = could not find container \"d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9\": container with ID starting with d9b198a78d43447b6076205b479c6314f894d745d4d28ac73fd577ff13fc19e9 not found: ID does not exist"
Jul 07 14:45:14.469016 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.468995    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"]
Jul 07 14:45:14.472964 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.472942    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f31ce-predictor-5f44c97f5c-k6ccb"]
Jul 07 14:45:14.509092 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.509056    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn4ct\" (UniqueName: \"kubernetes.io/projected/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-api-access-wn4ct\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:45:14.509092 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.509086    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6f46bc9-feae-4385-8cda-0f73e34675fa-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:45:14.509092 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:14.509099    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6f46bc9-feae-4385-8cda-0f73e34675fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:45:15.485393 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:15.485359    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" path="/var/lib/kubelet/pods/f6f46bc9-feae-4385-8cda-0f73e34675fa/volumes"
Jul 07 14:45:18.580850 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:18.580794    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:19.450178 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:19.450146    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:45:19.450716 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:19.450691    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:45:21.314104 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.314067    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"]
Jul 07 14:45:21.314525 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.314365    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kube-rbac-proxy"
Jul 07 14:45:21.314525 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.314376    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kube-rbac-proxy"
Jul 07 14:45:21.314525 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.314393    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container"
Jul 07 14:45:21.314525 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.314398    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container"
Jul 07 14:45:21.315734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.315019    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kube-rbac-proxy"
Jul 07 14:45:21.315734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.315050    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f46bc9-feae-4385-8cda-0f73e34675fa" containerName="kserve-container"
Jul 07 14:45:21.320361 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.320333    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:21.322940 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.322912    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-6e67f-kube-rbac-proxy-sar-config\""
Jul 07 14:45:21.323059 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.322947    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-6e67f-serving-cert\""
Jul 07 14:45:21.329478 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.329451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"]
Jul 07 14:45:21.359179 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.359139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20185bb-2a2c-4f60-8fbe-90e80732d150-openshift-service-ca-bundle\") pod \"splitter-graph-6e67f-fcb647484-fllj6\" (UID: \"d20185bb-2a2c-4f60-8fbe-90e80732d150\") " pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:21.359179 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.359185    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d20185bb-2a2c-4f60-8fbe-90e80732d150-proxy-tls\") pod \"splitter-graph-6e67f-fcb647484-fllj6\" (UID: \"d20185bb-2a2c-4f60-8fbe-90e80732d150\") " pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:21.460097 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.460057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20185bb-2a2c-4f60-8fbe-90e80732d150-openshift-service-ca-bundle\") pod \"splitter-graph-6e67f-fcb647484-fllj6\" (UID: \"d20185bb-2a2c-4f60-8fbe-90e80732d150\") " pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:21.460273 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.460105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d20185bb-2a2c-4f60-8fbe-90e80732d150-proxy-tls\") pod \"splitter-graph-6e67f-fcb647484-fllj6\" (UID: \"d20185bb-2a2c-4f60-8fbe-90e80732d150\") " pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:21.460273 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:21.460257    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-6e67f-serving-cert: secret "splitter-graph-6e67f-serving-cert" not found
Jul 07 14:45:21.460362 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:21.460338    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d20185bb-2a2c-4f60-8fbe-90e80732d150-proxy-tls podName:d20185bb-2a2c-4f60-8fbe-90e80732d150 nodeName:}" failed. No retries permitted until 2026-07-07 14:45:21.960314838 +0000 UTC m=+2172.957905533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d20185bb-2a2c-4f60-8fbe-90e80732d150-proxy-tls") pod "splitter-graph-6e67f-fcb647484-fllj6" (UID: "d20185bb-2a2c-4f60-8fbe-90e80732d150") : secret "splitter-graph-6e67f-serving-cert" not found
Jul 07 14:45:21.460833 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.460816    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20185bb-2a2c-4f60-8fbe-90e80732d150-openshift-service-ca-bundle\") pod \"splitter-graph-6e67f-fcb647484-fllj6\" (UID: \"d20185bb-2a2c-4f60-8fbe-90e80732d150\") " pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:21.964159 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.964122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d20185bb-2a2c-4f60-8fbe-90e80732d150-proxy-tls\") pod \"splitter-graph-6e67f-fcb647484-fllj6\" (UID: \"d20185bb-2a2c-4f60-8fbe-90e80732d150\") " pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:21.966649 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:21.966596    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d20185bb-2a2c-4f60-8fbe-90e80732d150-proxy-tls\") pod \"splitter-graph-6e67f-fcb647484-fllj6\" (UID: \"d20185bb-2a2c-4f60-8fbe-90e80732d150\") " pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:22.231765 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:22.231678    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:22.364913 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:22.364878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"]
Jul 07 14:45:22.368527 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:45:22.368500    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20185bb_2a2c_4f60_8fbe_90e80732d150.slice/crio-48d92c2185e4fa59a620e4443ea8a244ff8ec779b1ef38c7a566dd02fe6c8ad0 WatchSource:0}: Error finding container 48d92c2185e4fa59a620e4443ea8a244ff8ec779b1ef38c7a566dd02fe6c8ad0: Status 404 returned error can't find the container with id 48d92c2185e4fa59a620e4443ea8a244ff8ec779b1ef38c7a566dd02fe6c8ad0
Jul 07 14:45:22.477549 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:22.477513    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" event={"ID":"d20185bb-2a2c-4f60-8fbe-90e80732d150","Type":"ContainerStarted","Data":"5609ec5aa6923e5baec58eade1279c99010c71f62da6dc76976daee97d5e30ab"}
Jul 07 14:45:22.477714 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:22.477573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" event={"ID":"d20185bb-2a2c-4f60-8fbe-90e80732d150","Type":"ContainerStarted","Data":"48d92c2185e4fa59a620e4443ea8a244ff8ec779b1ef38c7a566dd02fe6c8ad0"}
Jul 07 14:45:22.477780 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:22.477718    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:22.494093 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:22.493979    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" podStartSLOduration=1.4939556330000001 podStartE2EDuration="1.493955633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:45:22.493512623 +0000 UTC m=+2173.491103340" watchObservedRunningTime="2026-07-07 14:45:22.493955633 +0000 UTC m=+2173.491546351"
Jul 07 14:45:23.581423 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:23.581386    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:23.581831 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:23.581494    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:45:28.488808 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:28.488780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:28.580898 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:28.580853    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:29.451729 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:29.451685    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:45:31.400419 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.400387    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"]
Jul 07 14:45:31.401012 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.400975    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f" containerID="cri-o://5609ec5aa6923e5baec58eade1279c99010c71f62da6dc76976daee97d5e30ab" gracePeriod=30
Jul 07 14:45:31.512267 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.512123    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"]
Jul 07 14:45:31.512455 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.512398    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container" containerID="cri-o://420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea" gracePeriod=30
Jul 07 14:45:31.512534 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.512506    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kube-rbac-proxy" containerID="cri-o://b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152" gracePeriod=30
Jul 07 14:45:31.538823 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.538793    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"]
Jul 07 14:45:31.542461 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.542436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:31.544688 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.544662    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7beb0-kube-rbac-proxy-sar-config\""
Jul 07 14:45:31.544819 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.544662    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7beb0-predictor-serving-cert\""
Jul 07 14:45:31.551931 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.551905    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"]
Jul 07 14:45:31.643556 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.643520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkxc\" (UniqueName: \"kubernetes.io/projected/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-api-access-grkxc\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:31.643750 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.643581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:31.643750 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.643640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245eaf0e-81d9-4381-a501-33bb7f9952ef-proxy-tls\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:31.745001 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.744917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkxc\" (UniqueName: \"kubernetes.io/projected/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-api-access-grkxc\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:31.745001 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.744957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:31.745001 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.744980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245eaf0e-81d9-4381-a501-33bb7f9952ef-proxy-tls\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:31.745272 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:31.745093    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-serving-cert: secret "success-200-isvc-7beb0-predictor-serving-cert" not found
Jul 07 14:45:31.745272 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:31.745148    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245eaf0e-81d9-4381-a501-33bb7f9952ef-proxy-tls podName:245eaf0e-81d9-4381-a501-33bb7f9952ef nodeName:}" failed. No retries permitted until 2026-07-07 14:45:32.245126946 +0000 UTC m=+2183.242717658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/245eaf0e-81d9-4381-a501-33bb7f9952ef-proxy-tls") pod "success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" (UID: "245eaf0e-81d9-4381-a501-33bb7f9952ef") : secret "success-200-isvc-7beb0-predictor-serving-cert" not found
Jul 07 14:45:31.745568 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.745549    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:31.753385 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:31.753362    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkxc\" (UniqueName: \"kubernetes.io/projected/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-api-access-grkxc\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:32.249317 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:32.249277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245eaf0e-81d9-4381-a501-33bb7f9952ef-proxy-tls\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:32.251815 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:32.251780    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245eaf0e-81d9-4381-a501-33bb7f9952ef-proxy-tls\") pod \"success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") " pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:32.452635 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:32.452593    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:32.523537 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:32.523506    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerID="b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152" exitCode=2
Jul 07 14:45:32.523744 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:32.523583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" event={"ID":"7f271e4c-1439-419a-8b46-04b1c7b34595","Type":"ContainerDied","Data":"b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152"}
Jul 07 14:45:32.590395 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:32.590193    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"]
Jul 07 14:45:32.592950 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:45:32.592925    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245eaf0e_81d9_4381_a501_33bb7f9952ef.slice/crio-8c55a964ff3b84e4d1cf72361b9bf159e9270070de7b5f4ce8382d76eadd77b2 WatchSource:0}: Error finding container 8c55a964ff3b84e4d1cf72361b9bf159e9270070de7b5f4ce8382d76eadd77b2: Status 404 returned error can't find the container with id 8c55a964ff3b84e4d1cf72361b9bf159e9270070de7b5f4ce8382d76eadd77b2
Jul 07 14:45:33.487470 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:33.487434    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:33.530772 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:33.530738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" event={"ID":"245eaf0e-81d9-4381-a501-33bb7f9952ef","Type":"ContainerStarted","Data":"a94c1727b28ca61163f41653e70220d7d1ab6c26e08e642ede5c97f8f3ab1be7"}
Jul 07 14:45:33.530968 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:33.530813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" event={"ID":"245eaf0e-81d9-4381-a501-33bb7f9952ef","Type":"ContainerStarted","Data":"8e80a55ddb4077230098b45237ffc81c398a0000973b5d4abdfc4a6d17c6cb64"}
Jul 07 14:45:33.530968 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:33.530843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" event={"ID":"245eaf0e-81d9-4381-a501-33bb7f9952ef","Type":"ContainerStarted","Data":"8c55a964ff3b84e4d1cf72361b9bf159e9270070de7b5f4ce8382d76eadd77b2"}
Jul 07 14:45:33.530968 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:33.530875    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:33.549582 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:33.549529    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" podStartSLOduration=2.54950936 podStartE2EDuration="2.54950936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:45:33.548322746 +0000 UTC m=+2184.545913453" watchObservedRunningTime="2026-07-07 14:45:33.54950936 +0000 UTC m=+2184.547100074"
Jul 07 14:45:33.581330 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:33.581284    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:34.534674 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:34.534643    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:34.536050 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:34.536022    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 14:45:34.954555 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:34.954532    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:45:35.071744 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.071652    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlzlh\" (UniqueName: \"kubernetes.io/projected/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-api-access-nlzlh\") pod \"7f271e4c-1439-419a-8b46-04b1c7b34595\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") "
Jul 07 14:45:35.071744 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.071699    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f271e4c-1439-419a-8b46-04b1c7b34595-proxy-tls\") pod \"7f271e4c-1439-419a-8b46-04b1c7b34595\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") "
Jul 07 14:45:35.071955 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.071754    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-rbac-proxy-sar-config\") pod \"7f271e4c-1439-419a-8b46-04b1c7b34595\" (UID: \"7f271e4c-1439-419a-8b46-04b1c7b34595\") "
Jul 07 14:45:35.072131 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.072109    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f271e4c-1439-419a-8b46-04b1c7b34595" (UID: "7f271e4c-1439-419a-8b46-04b1c7b34595"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:45:35.073794 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.073768    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-api-access-nlzlh" (OuterVolumeSpecName: "kube-api-access-nlzlh") pod "7f271e4c-1439-419a-8b46-04b1c7b34595" (UID: "7f271e4c-1439-419a-8b46-04b1c7b34595"). InnerVolumeSpecName "kube-api-access-nlzlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:45:35.073875 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.073795    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f271e4c-1439-419a-8b46-04b1c7b34595-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f271e4c-1439-419a-8b46-04b1c7b34595" (UID: "7f271e4c-1439-419a-8b46-04b1c7b34595"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:45:35.172866 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.172823    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f271e4c-1439-419a-8b46-04b1c7b34595-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:45:35.172866 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.172859    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:45:35.172866 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.172869    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzlh\" (UniqueName: \"kubernetes.io/projected/7f271e4c-1439-419a-8b46-04b1c7b34595-kube-api-access-nlzlh\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:45:35.540281 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.540244    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerID="420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea" exitCode=0
Jul 07 14:45:35.540692 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.540291    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"
Jul 07 14:45:35.540692 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.540319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" event={"ID":"7f271e4c-1439-419a-8b46-04b1c7b34595","Type":"ContainerDied","Data":"420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea"}
Jul 07 14:45:35.540692 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.540377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj" event={"ID":"7f271e4c-1439-419a-8b46-04b1c7b34595","Type":"ContainerDied","Data":"9df1dd4d76138695cb2e2f2e23ad2328e0d352c6d486b1a658058bbbb4569bc0"}
Jul 07 14:45:35.540692 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.540403    2565 scope.go:117] "RemoveContainer" containerID="b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152"
Jul 07 14:45:35.541016 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.540993    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 14:45:35.549163 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.549126    2565 scope.go:117] "RemoveContainer" containerID="420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea"
Jul 07 14:45:35.559766 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.559559    2565 scope.go:117] "RemoveContainer" containerID="b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152"
Jul 07 14:45:35.560154 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:35.560126    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152\": container with ID starting with b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152 not found: ID does not exist" containerID="b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152"
Jul 07 14:45:35.560278 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.560166    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152"} err="failed to get container status \"b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152\": rpc error: code = NotFound desc = could not find container \"b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152\": container with ID starting with b91bcdfec5de491a2aa06e133b62c07abec638a3d011707e0c604f9deb748152 not found: ID does not exist"
Jul 07 14:45:35.560278 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.560185    2565 scope.go:117] "RemoveContainer" containerID="420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea"
Jul 07 14:45:35.560520 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:35.560498    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea\": container with ID starting with 420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea not found: ID does not exist" containerID="420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea"
Jul 07 14:45:35.560584 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.560525    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea"} err="failed to get container status \"420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea\": rpc error: code = NotFound desc = could not find container \"420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea\": container with ID starting with 420f402c70cae3eb4926bcf588b0031a740e4449723a2685cae9ec186df4fbea not found: ID does not exist"
Jul 07 14:45:35.561765 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.561740    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"]
Jul 07 14:45:35.566911 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:35.566880    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e67f-predictor-787f79cf5c-wnvpj"]
Jul 07 14:45:37.487505 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:37.487464    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" path="/var/lib/kubelet/pods/7f271e4c-1439-419a-8b46-04b1c7b34595/volumes"
Jul 07 14:45:38.486648 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:38.486598    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:38.581926 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:38.581884    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:39.451521 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:39.451479    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:45:40.545691 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:40.545662    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:45:40.546081 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:40.546058    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 14:45:40.992793 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:40.992769    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:45:41.120659 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.120610    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b2a024a-5068-40c0-8124-bc69b169e130-proxy-tls\") pod \"4b2a024a-5068-40c0-8124-bc69b169e130\" (UID: \"4b2a024a-5068-40c0-8124-bc69b169e130\") "
Jul 07 14:45:41.120808 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.120698    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b2a024a-5068-40c0-8124-bc69b169e130-openshift-service-ca-bundle\") pod \"4b2a024a-5068-40c0-8124-bc69b169e130\" (UID: \"4b2a024a-5068-40c0-8124-bc69b169e130\") "
Jul 07 14:45:41.121095 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.121061    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2a024a-5068-40c0-8124-bc69b169e130-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4b2a024a-5068-40c0-8124-bc69b169e130" (UID: "4b2a024a-5068-40c0-8124-bc69b169e130"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:45:41.122752 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.122731    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2a024a-5068-40c0-8124-bc69b169e130-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b2a024a-5068-40c0-8124-bc69b169e130" (UID: "4b2a024a-5068-40c0-8124-bc69b169e130"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:45:41.222117 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.222075    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b2a024a-5068-40c0-8124-bc69b169e130-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:45:41.222117 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.222108    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b2a024a-5068-40c0-8124-bc69b169e130-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:45:41.565907 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.565871    2565 generic.go:356] "Generic (PLEG): container finished" podID="4b2a024a-5068-40c0-8124-bc69b169e130" containerID="508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450" exitCode=0
Jul 07 14:45:41.566365 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.565917    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"
Jul 07 14:45:41.566365 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.565955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" event={"ID":"4b2a024a-5068-40c0-8124-bc69b169e130","Type":"ContainerDied","Data":"508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450"}
Jul 07 14:45:41.566365 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.566006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8" event={"ID":"4b2a024a-5068-40c0-8124-bc69b169e130","Type":"ContainerDied","Data":"22401f3e28d10ec0366d023f068e4bc8c3f1de1a57298f7622c8f9545545be79"}
Jul 07 14:45:41.566365 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.566042    2565 scope.go:117] "RemoveContainer" containerID="508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450"
Jul 07 14:45:41.574655 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.574611    2565 scope.go:117] "RemoveContainer" containerID="508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450"
Jul 07 14:45:41.575005 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:45:41.574982    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450\": container with ID starting with 508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450 not found: ID does not exist" containerID="508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450"
Jul 07 14:45:41.575097 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.575024    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450"} err="failed to get container status \"508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450\": rpc error: code = NotFound desc = could not find container \"508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450\": container with ID starting with 508f80ccb7af3db936253d43fb1a9fb7b555c7f3bd3446626c2b201eb5d80450 not found: ID does not exist"
Jul 07 14:45:41.583454 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.583421    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"]
Jul 07 14:45:41.586985 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:41.586953    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f31ce-9c588fcbb-qjxf8"]
Jul 07 14:45:43.486761 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:43.486726    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:43.487137 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:43.486944    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" path="/var/lib/kubelet/pods/4b2a024a-5068-40c0-8124-bc69b169e130/volumes"
Jul 07 14:45:43.487234 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:43.487222    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:45:48.486765 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:48.486727    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:49.451092 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:49.451050    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 07 14:45:50.546846 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:50.546807    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 14:45:53.486850 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:53.486805    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:58.487215 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:58.487169    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:45:59.451806 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:45:59.451775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 14:46:00.546831 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:00.546786    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 14:46:01.653104 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:01.653066    2565 generic.go:356] "Generic (PLEG): container finished" podID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerID="5609ec5aa6923e5baec58eade1279c99010c71f62da6dc76976daee97d5e30ab" exitCode=0
Jul 07 14:46:01.653475 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:01.653140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" event={"ID":"d20185bb-2a2c-4f60-8fbe-90e80732d150","Type":"ContainerDied","Data":"5609ec5aa6923e5baec58eade1279c99010c71f62da6dc76976daee97d5e30ab"}
Jul 07 14:46:02.052151 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.052122    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:46:02.196130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.196033    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d20185bb-2a2c-4f60-8fbe-90e80732d150-proxy-tls\") pod \"d20185bb-2a2c-4f60-8fbe-90e80732d150\" (UID: \"d20185bb-2a2c-4f60-8fbe-90e80732d150\") "
Jul 07 14:46:02.196130 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.196078    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20185bb-2a2c-4f60-8fbe-90e80732d150-openshift-service-ca-bundle\") pod \"d20185bb-2a2c-4f60-8fbe-90e80732d150\" (UID: \"d20185bb-2a2c-4f60-8fbe-90e80732d150\") "
Jul 07 14:46:02.196469 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.196439    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20185bb-2a2c-4f60-8fbe-90e80732d150-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d20185bb-2a2c-4f60-8fbe-90e80732d150" (UID: "d20185bb-2a2c-4f60-8fbe-90e80732d150"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:46:02.198248 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.198224    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20185bb-2a2c-4f60-8fbe-90e80732d150-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d20185bb-2a2c-4f60-8fbe-90e80732d150" (UID: "d20185bb-2a2c-4f60-8fbe-90e80732d150"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:46:02.297324 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.297285    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d20185bb-2a2c-4f60-8fbe-90e80732d150-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:46:02.297324 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.297317    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d20185bb-2a2c-4f60-8fbe-90e80732d150-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:46:02.658114 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.658082    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"
Jul 07 14:46:02.658567 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.658109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6" event={"ID":"d20185bb-2a2c-4f60-8fbe-90e80732d150","Type":"ContainerDied","Data":"48d92c2185e4fa59a620e4443ea8a244ff8ec779b1ef38c7a566dd02fe6c8ad0"}
Jul 07 14:46:02.658567 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.658176    2565 scope.go:117] "RemoveContainer" containerID="5609ec5aa6923e5baec58eade1279c99010c71f62da6dc76976daee97d5e30ab"
Jul 07 14:46:02.682246 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.682216    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"]
Jul 07 14:46:02.684949 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:02.684919    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6e67f-fcb647484-fllj6"]
Jul 07 14:46:03.486766 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:03.486731    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" path="/var/lib/kubelet/pods/d20185bb-2a2c-4f60-8fbe-90e80732d150/volumes"
Jul 07 14:46:10.546857 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:10.546817    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 07 14:46:11.034824 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.034784    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"]
Jul 07 14:46:11.035125 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035113    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce"
Jul 07 14:46:11.035170 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035127    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce"
Jul 07 14:46:11.035170 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035162    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container"
Jul 07 14:46:11.035170 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035168    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container"
Jul 07 14:46:11.035263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035180    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f"
Jul 07 14:46:11.035263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035186    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f"
Jul 07 14:46:11.035263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035196    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kube-rbac-proxy"
Jul 07 14:46:11.035263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035201    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kube-rbac-proxy"
Jul 07 14:46:11.035263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035255    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2a024a-5068-40c0-8124-bc69b169e130" containerName="sequence-graph-f31ce"
Jul 07 14:46:11.035263 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035262    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20185bb-2a2c-4f60-8fbe-90e80732d150" containerName="splitter-graph-6e67f"
Jul 07 14:46:11.035433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035268    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kube-rbac-proxy"
Jul 07 14:46:11.035433 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.035275    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f271e4c-1439-419a-8b46-04b1c7b34595" containerName="kserve-container"
Jul 07 14:46:11.039831 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.039810    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:11.042162 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.042131    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-36f00-serving-cert\""
Jul 07 14:46:11.042310 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.042282    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-36f00-kube-rbac-proxy-sar-config\""
Jul 07 14:46:11.047450 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.047421    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"]
Jul 07 14:46:11.167198 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.167156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d3e8bd-7666-48aa-9b00-a9523ace4547-openshift-service-ca-bundle\") pod \"switch-graph-36f00-7bc7bf4797-g5dbt\" (UID: \"20d3e8bd-7666-48aa-9b00-a9523ace4547\") " pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:11.167198 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.167201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20d3e8bd-7666-48aa-9b00-a9523ace4547-proxy-tls\") pod \"switch-graph-36f00-7bc7bf4797-g5dbt\" (UID: \"20d3e8bd-7666-48aa-9b00-a9523ace4547\") " pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:11.268368 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.268324    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d3e8bd-7666-48aa-9b00-a9523ace4547-openshift-service-ca-bundle\") pod \"switch-graph-36f00-7bc7bf4797-g5dbt\" (UID: \"20d3e8bd-7666-48aa-9b00-a9523ace4547\") " pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:11.268368 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.268368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20d3e8bd-7666-48aa-9b00-a9523ace4547-proxy-tls\") pod \"switch-graph-36f00-7bc7bf4797-g5dbt\" (UID: \"20d3e8bd-7666-48aa-9b00-a9523ace4547\") " pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:11.268749 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:46:11.268501    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-36f00-serving-cert: secret "switch-graph-36f00-serving-cert" not found
Jul 07 14:46:11.268749 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:46:11.268574    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d3e8bd-7666-48aa-9b00-a9523ace4547-proxy-tls podName:20d3e8bd-7666-48aa-9b00-a9523ace4547 nodeName:}" failed. No retries permitted until 2026-07-07 14:46:11.768551475 +0000 UTC m=+2222.766142181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/20d3e8bd-7666-48aa-9b00-a9523ace4547-proxy-tls") pod "switch-graph-36f00-7bc7bf4797-g5dbt" (UID: "20d3e8bd-7666-48aa-9b00-a9523ace4547") : secret "switch-graph-36f00-serving-cert" not found
Jul 07 14:46:11.269004 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.268982    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d3e8bd-7666-48aa-9b00-a9523ace4547-openshift-service-ca-bundle\") pod \"switch-graph-36f00-7bc7bf4797-g5dbt\" (UID: \"20d3e8bd-7666-48aa-9b00-a9523ace4547\") " pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:11.773462 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.773426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20d3e8bd-7666-48aa-9b00-a9523ace4547-proxy-tls\") pod \"switch-graph-36f00-7bc7bf4797-g5dbt\" (UID: \"20d3e8bd-7666-48aa-9b00-a9523ace4547\") " pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:11.775970 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.775949    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20d3e8bd-7666-48aa-9b00-a9523ace4547-proxy-tls\") pod \"switch-graph-36f00-7bc7bf4797-g5dbt\" (UID: \"20d3e8bd-7666-48aa-9b00-a9523ace4547\") " pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:11.951545 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:11.951508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:12.081013 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:12.080978    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"]
Jul 07 14:46:12.084390 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:46:12.084363    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d3e8bd_7666_48aa_9b00_a9523ace4547.slice/crio-9fb2246c7aebfe363fe75c18c17ea47a68d1214a1c275774c1307992fc63837b WatchSource:0}: Error finding container 9fb2246c7aebfe363fe75c18c17ea47a68d1214a1c275774c1307992fc63837b: Status 404 returned error can't find the container with id 9fb2246c7aebfe363fe75c18c17ea47a68d1214a1c275774c1307992fc63837b
Jul 07 14:46:12.700519 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:12.700487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" event={"ID":"20d3e8bd-7666-48aa-9b00-a9523ace4547","Type":"ContainerStarted","Data":"70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75"}
Jul 07 14:46:12.700838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:12.700547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" event={"ID":"20d3e8bd-7666-48aa-9b00-a9523ace4547","Type":"ContainerStarted","Data":"9fb2246c7aebfe363fe75c18c17ea47a68d1214a1c275774c1307992fc63837b"}
Jul 07 14:46:12.700838 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:12.700608    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:12.718875 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:12.718820    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" podStartSLOduration=1.718801995 podStartE2EDuration="1.718801995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:46:12.717268158 +0000 UTC m=+2223.714858885" watchObservedRunningTime="2026-07-07 14:46:12.718801995 +0000 UTC m=+2223.716392710"
Jul 07 14:46:18.709328 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:18.709298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 14:46:20.547438 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:20.547410    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:46:31.605197 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.605157    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"]
Jul 07 14:46:31.608503 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.608485    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:31.610609 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.610583    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7beb0-serving-cert\""
Jul 07 14:46:31.610764 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.610669    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7beb0-kube-rbac-proxy-sar-config\""
Jul 07 14:46:31.616117 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.616095    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"]
Jul 07 14:46:31.627850 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.627821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-openshift-service-ca-bundle\") pod \"splitter-graph-7beb0-6cc4b5b5bc-brf96\" (UID: \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\") " pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:31.628013 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.627884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-proxy-tls\") pod \"splitter-graph-7beb0-6cc4b5b5bc-brf96\" (UID: \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\") " pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:31.728726 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.728689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-openshift-service-ca-bundle\") pod \"splitter-graph-7beb0-6cc4b5b5bc-brf96\" (UID: \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\") " pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:31.728906 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.728744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-proxy-tls\") pod \"splitter-graph-7beb0-6cc4b5b5bc-brf96\" (UID: \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\") " pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:31.728906 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:46:31.728869    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-7beb0-serving-cert: secret "splitter-graph-7beb0-serving-cert" not found
Jul 07 14:46:31.728976 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:46:31.728926    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-proxy-tls podName:f4d40633-686a-4b6b-a8fa-b4cabeff4f57 nodeName:}" failed. No retries permitted until 2026-07-07 14:46:32.228909617 +0000 UTC m=+2243.226500310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-proxy-tls") pod "splitter-graph-7beb0-6cc4b5b5bc-brf96" (UID: "f4d40633-686a-4b6b-a8fa-b4cabeff4f57") : secret "splitter-graph-7beb0-serving-cert" not found
Jul 07 14:46:31.729361 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:31.729342    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-openshift-service-ca-bundle\") pod \"splitter-graph-7beb0-6cc4b5b5bc-brf96\" (UID: \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\") " pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:32.232857 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:32.232820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-proxy-tls\") pod \"splitter-graph-7beb0-6cc4b5b5bc-brf96\" (UID: \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\") " pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:32.235369 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:32.235333    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-proxy-tls\") pod \"splitter-graph-7beb0-6cc4b5b5bc-brf96\" (UID: \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\") " pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:32.520006 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:32.519910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:32.648734 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:32.648703    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"]
Jul 07 14:46:32.651159 ip-10-0-138-82 kubenswrapper[2565]: W0707 14:46:32.651132    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d40633_686a_4b6b_a8fa_b4cabeff4f57.slice/crio-7cca2f2ac9e44b557afda7fbb3c75bee8dfa00eb7ee5ff55ab8b32a3783d086b WatchSource:0}: Error finding container 7cca2f2ac9e44b557afda7fbb3c75bee8dfa00eb7ee5ff55ab8b32a3783d086b: Status 404 returned error can't find the container with id 7cca2f2ac9e44b557afda7fbb3c75bee8dfa00eb7ee5ff55ab8b32a3783d086b
Jul 07 14:46:32.777424 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:32.777329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" event={"ID":"f4d40633-686a-4b6b-a8fa-b4cabeff4f57","Type":"ContainerStarted","Data":"bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf"}
Jul 07 14:46:32.777424 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:32.777394    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" event={"ID":"f4d40633-686a-4b6b-a8fa-b4cabeff4f57","Type":"ContainerStarted","Data":"7cca2f2ac9e44b557afda7fbb3c75bee8dfa00eb7ee5ff55ab8b32a3783d086b"}
Jul 07 14:46:32.777644 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:32.777444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:46:32.796216 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:32.796143    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" podStartSLOduration=1.796121541 podStartE2EDuration="1.796121541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:46:32.795818669 +0000 UTC m=+2243.793409384" watchObservedRunningTime="2026-07-07 14:46:32.796121541 +0000 UTC m=+2243.793712259"
Jul 07 14:46:38.787907 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:46:38.787870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:54:46.307722 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:46.307684    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"]
Jul 07 14:54:46.308281 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:46.307886    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0" containerID="cri-o://bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf" gracePeriod=30
Jul 07 14:54:46.484666 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:46.484633    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"]
Jul 07 14:54:46.484923 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:46.484886    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container" containerID="cri-o://8e80a55ddb4077230098b45237ffc81c398a0000973b5d4abdfc4a6d17c6cb64" gracePeriod=30
Jul 07 14:54:46.485057 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:46.484908    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kube-rbac-proxy" containerID="cri-o://a94c1727b28ca61163f41653e70220d7d1ab6c26e08e642ede5c97f8f3ab1be7" gracePeriod=30
Jul 07 14:54:46.632062 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:46.632027    2565 generic.go:356] "Generic (PLEG): container finished" podID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerID="a94c1727b28ca61163f41653e70220d7d1ab6c26e08e642ede5c97f8f3ab1be7" exitCode=2
Jul 07 14:54:46.632259 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:46.632100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" event={"ID":"245eaf0e-81d9-4381-a501-33bb7f9952ef","Type":"ContainerDied","Data":"a94c1727b28ca61163f41653e70220d7d1ab6c26e08e642ede5c97f8f3ab1be7"}
Jul 07 14:54:48.785869 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:48.785833    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:54:49.646177 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.646145    2565 generic.go:356] "Generic (PLEG): container finished" podID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerID="8e80a55ddb4077230098b45237ffc81c398a0000973b5d4abdfc4a6d17c6cb64" exitCode=0
Jul 07 14:54:49.646333 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.646216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" event={"ID":"245eaf0e-81d9-4381-a501-33bb7f9952ef","Type":"ContainerDied","Data":"8e80a55ddb4077230098b45237ffc81c398a0000973b5d4abdfc4a6d17c6cb64"}
Jul 07 14:54:49.739208 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.739181    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:54:49.824084 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.824051    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grkxc\" (UniqueName: \"kubernetes.io/projected/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-api-access-grkxc\") pod \"245eaf0e-81d9-4381-a501-33bb7f9952ef\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") "
Jul 07 14:54:49.824520 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.824092    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-rbac-proxy-sar-config\") pod \"245eaf0e-81d9-4381-a501-33bb7f9952ef\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") "
Jul 07 14:54:49.824520 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.824154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245eaf0e-81d9-4381-a501-33bb7f9952ef-proxy-tls\") pod \"245eaf0e-81d9-4381-a501-33bb7f9952ef\" (UID: \"245eaf0e-81d9-4381-a501-33bb7f9952ef\") "
Jul 07 14:54:49.824596 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.824538    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "245eaf0e-81d9-4381-a501-33bb7f9952ef" (UID: "245eaf0e-81d9-4381-a501-33bb7f9952ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:54:49.826212 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.826179    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245eaf0e-81d9-4381-a501-33bb7f9952ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "245eaf0e-81d9-4381-a501-33bb7f9952ef" (UID: "245eaf0e-81d9-4381-a501-33bb7f9952ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:54:49.826354 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.826254    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-api-access-grkxc" (OuterVolumeSpecName: "kube-api-access-grkxc") pod "245eaf0e-81d9-4381-a501-33bb7f9952ef" (UID: "245eaf0e-81d9-4381-a501-33bb7f9952ef"). InnerVolumeSpecName "kube-api-access-grkxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 14:54:49.925179 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.925138    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grkxc\" (UniqueName: \"kubernetes.io/projected/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-api-access-grkxc\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:54:49.925179 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.925169    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/245eaf0e-81d9-4381-a501-33bb7f9952ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:54:49.925179 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:49.925178    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/245eaf0e-81d9-4381-a501-33bb7f9952ef-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:54:50.651991 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:50.651956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6" event={"ID":"245eaf0e-81d9-4381-a501-33bb7f9952ef","Type":"ContainerDied","Data":"8c55a964ff3b84e4d1cf72361b9bf159e9270070de7b5f4ce8382d76eadd77b2"}
Jul 07 14:54:50.652198 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:50.652001    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"
Jul 07 14:54:50.652198 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:50.652034    2565 scope.go:117] "RemoveContainer" containerID="a94c1727b28ca61163f41653e70220d7d1ab6c26e08e642ede5c97f8f3ab1be7"
Jul 07 14:54:50.660891 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:50.660873    2565 scope.go:117] "RemoveContainer" containerID="8e80a55ddb4077230098b45237ffc81c398a0000973b5d4abdfc4a6d17c6cb64"
Jul 07 14:54:50.676739 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:50.676708    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"]
Jul 07 14:54:50.681466 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:50.681437    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7beb0-predictor-5d5b878bd9-dbpz6"]
Jul 07 14:54:51.485536 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:51.485498    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" path="/var/lib/kubelet/pods/245eaf0e-81d9-4381-a501-33bb7f9952ef/volumes"
Jul 07 14:54:53.785680 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:53.785641    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:54:58.786259 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:58.786210    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:54:58.786780 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:54:58.786344    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:55:03.785148 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:03.785106    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:55:08.787119 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:08.787070    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:55:13.785323 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:13.785287    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 14:55:16.446918 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.446892    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:55:16.547372 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.547344    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-openshift-service-ca-bundle\") pod \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\" (UID: \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\") "
Jul 07 14:55:16.547557 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.547390    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-proxy-tls\") pod \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\" (UID: \"f4d40633-686a-4b6b-a8fa-b4cabeff4f57\") "
Jul 07 14:55:16.547755 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.547730    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f4d40633-686a-4b6b-a8fa-b4cabeff4f57" (UID: "f4d40633-686a-4b6b-a8fa-b4cabeff4f57"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 14:55:16.549422 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.549392    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4d40633-686a-4b6b-a8fa-b4cabeff4f57" (UID: "f4d40633-686a-4b6b-a8fa-b4cabeff4f57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 14:55:16.648806 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.648765    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:55:16.648806 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.648801    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4d40633-686a-4b6b-a8fa-b4cabeff4f57-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 14:55:16.750751 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.750716    2565 generic.go:356] "Generic (PLEG): container finished" podID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerID="bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf" exitCode=0
Jul 07 14:55:16.750916 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.750760    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"
Jul 07 14:55:16.750916 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.750805    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" event={"ID":"f4d40633-686a-4b6b-a8fa-b4cabeff4f57","Type":"ContainerDied","Data":"bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf"}
Jul 07 14:55:16.750916 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.750859    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96" event={"ID":"f4d40633-686a-4b6b-a8fa-b4cabeff4f57","Type":"ContainerDied","Data":"7cca2f2ac9e44b557afda7fbb3c75bee8dfa00eb7ee5ff55ab8b32a3783d086b"}
Jul 07 14:55:16.750916 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.750886    2565 scope.go:117] "RemoveContainer" containerID="bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf"
Jul 07 14:55:16.759236 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.759218    2565 scope.go:117] "RemoveContainer" containerID="bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf"
Jul 07 14:55:16.759503 ip-10-0-138-82 kubenswrapper[2565]: E0707 14:55:16.759483    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf\": container with ID starting with bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf not found: ID does not exist" containerID="bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf"
Jul 07 14:55:16.759597 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.759509    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf"} err="failed to get container status \"bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf\": rpc error: code = NotFound desc = could not find container \"bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf\": container with ID starting with bb2adadba702f4a0783b39d9689675f019df9e52aac98bed670b364886772ebf not found: ID does not exist"
Jul 07 14:55:16.774896 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.774862    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"]
Jul 07 14:55:16.783195 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:16.783168    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7beb0-6cc4b5b5bc-brf96"]
Jul 07 14:55:17.486453 ip-10-0-138-82 kubenswrapper[2565]: I0707 14:55:17.486424    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" path="/var/lib/kubelet/pods/f4d40633-686a-4b6b-a8fa-b4cabeff4f57/volumes"
Jul 07 15:02:30.134974 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:30.134942    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"]
Jul 07 15:02:30.137384 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:30.135182    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00" containerID="cri-o://70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75" gracePeriod=30
Jul 07 15:02:30.345181 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:30.345143    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"]
Jul 07 15:02:30.345412 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:30.345371    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container" containerID="cri-o://cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656" gracePeriod=30
Jul 07 15:02:30.345505 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:30.345393    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kube-rbac-proxy" containerID="cri-o://f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e" gracePeriod=30
Jul 07 15:02:31.340809 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:31.340775    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerID="f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e" exitCode=2
Jul 07 15:02:31.341179 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:31.340848    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" event={"ID":"3c525a2c-686b-4c1c-afd0-e826d769d80a","Type":"ContainerDied","Data":"f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e"}
Jul 07 15:02:33.499985 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.499958    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 15:02:33.657195 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.657155    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c525a2c-686b-4c1c-afd0-e826d769d80a-proxy-tls\") pod \"3c525a2c-686b-4c1c-afd0-e826d769d80a\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") "
Jul 07 15:02:33.657385 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.657231    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-rbac-proxy-sar-config\") pod \"3c525a2c-686b-4c1c-afd0-e826d769d80a\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") "
Jul 07 15:02:33.657385 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.657254    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6x7n\" (UniqueName: \"kubernetes.io/projected/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-api-access-l6x7n\") pod \"3c525a2c-686b-4c1c-afd0-e826d769d80a\" (UID: \"3c525a2c-686b-4c1c-afd0-e826d769d80a\") "
Jul 07 15:02:33.657645 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.657591    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c525a2c-686b-4c1c-afd0-e826d769d80a" (UID: "3c525a2c-686b-4c1c-afd0-e826d769d80a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:02:33.659322 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.659302    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c525a2c-686b-4c1c-afd0-e826d769d80a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c525a2c-686b-4c1c-afd0-e826d769d80a" (UID: "3c525a2c-686b-4c1c-afd0-e826d769d80a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:02:33.659471 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.659421    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-api-access-l6x7n" (OuterVolumeSpecName: "kube-api-access-l6x7n") pod "3c525a2c-686b-4c1c-afd0-e826d769d80a" (UID: "3c525a2c-686b-4c1c-afd0-e826d769d80a"). InnerVolumeSpecName "kube-api-access-l6x7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 15:02:33.708673 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.708611    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:02:33.758072 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.758034    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c525a2c-686b-4c1c-afd0-e826d769d80a-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 15:02:33.758072 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.758066    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 15:02:33.758072 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:33.758076    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6x7n\" (UniqueName: \"kubernetes.io/projected/3c525a2c-686b-4c1c-afd0-e826d769d80a-kube-api-access-l6x7n\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 15:02:34.352789 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.352758    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerID="cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656" exitCode=0
Jul 07 15:02:34.352991 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.352809    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"
Jul 07 15:02:34.352991 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.352850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" event={"ID":"3c525a2c-686b-4c1c-afd0-e826d769d80a","Type":"ContainerDied","Data":"cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656"}
Jul 07 15:02:34.352991 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.352922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk" event={"ID":"3c525a2c-686b-4c1c-afd0-e826d769d80a","Type":"ContainerDied","Data":"858206f349ad5ca0539a48faf6d283937db30f340d69a65085cff2feab05a5e4"}
Jul 07 15:02:34.352991 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.352959    2565 scope.go:117] "RemoveContainer" containerID="f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e"
Jul 07 15:02:34.363666 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.363547    2565 scope.go:117] "RemoveContainer" containerID="cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656"
Jul 07 15:02:34.371067 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.371049    2565 scope.go:117] "RemoveContainer" containerID="f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e"
Jul 07 15:02:34.371361 ip-10-0-138-82 kubenswrapper[2565]: E0707 15:02:34.371340    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e\": container with ID starting with f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e not found: ID does not exist" containerID="f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e"
Jul 07 15:02:34.371410 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.371370    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e"} err="failed to get container status \"f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e\": rpc error: code = NotFound desc = could not find container \"f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e\": container with ID starting with f26019950fc2b23d41bf7e3d37addd556c8bd663270a6e2bb0439ffa12e4ea1e not found: ID does not exist"
Jul 07 15:02:34.371410 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.371388    2565 scope.go:117] "RemoveContainer" containerID="cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656"
Jul 07 15:02:34.371609 ip-10-0-138-82 kubenswrapper[2565]: E0707 15:02:34.371590    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656\": container with ID starting with cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656 not found: ID does not exist" containerID="cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656"
Jul 07 15:02:34.371715 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.371696    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656"} err="failed to get container status \"cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656\": rpc error: code = NotFound desc = could not find container \"cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656\": container with ID starting with cbfd80541bf40fcbe3ae5f5d200a60004af9eb529aad95d03e54f5365242f656 not found: ID does not exist"
Jul 07 15:02:34.380529 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.380501    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"]
Jul 07 15:02:34.386294 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:34.386266    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-36f00-predictor-869b59995f-knmkk"]
Jul 07 15:02:35.485676 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:35.485643    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" path="/var/lib/kubelet/pods/3c525a2c-686b-4c1c-afd0-e826d769d80a/volumes"
Jul 07 15:02:38.708114 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:38.708071    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:02:43.707806 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:43.707764    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:02:43.708306 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:43.707905    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 15:02:48.708113 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:48.708072    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:02:48.823786 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:48.823758    2565 ???:1] "http: TLS handshake error from 10.0.138.82:37556: EOF"
Jul 07 15:02:48.831362 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:48.831321    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:49.718886 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:49.718848    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:50.573114 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:50.573083    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:51.422360 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:51.422331    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:52.245383 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:52.245354    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:53.117112 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:53.117087    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:53.708554 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:53.708517    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:02:54.008952 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:54.008858    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:54.851270 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:54.851229    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:55.745261 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:55.745233    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:56.640848 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:56.640819    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:57.479948 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:57.479911    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:58.357496 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:58.357466    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:02:58.708107 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:58.708064    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 15:02:59.257050 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:02:59.257013    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:03:00.207201 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.207173    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-36f00-7bc7bf4797-g5dbt_20d3e8bd-7666-48aa-9b00-a9523ace4547/switch-graph-36f00/0.log"
Jul 07 15:03:00.290188 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.290162    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 15:03:00.349226 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.349187    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20d3e8bd-7666-48aa-9b00-a9523ace4547-proxy-tls\") pod \"20d3e8bd-7666-48aa-9b00-a9523ace4547\" (UID: \"20d3e8bd-7666-48aa-9b00-a9523ace4547\") "
Jul 07 15:03:00.349426 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.349274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d3e8bd-7666-48aa-9b00-a9523ace4547-openshift-service-ca-bundle\") pod \"20d3e8bd-7666-48aa-9b00-a9523ace4547\" (UID: \"20d3e8bd-7666-48aa-9b00-a9523ace4547\") "
Jul 07 15:03:00.349674 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.349612    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d3e8bd-7666-48aa-9b00-a9523ace4547-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "20d3e8bd-7666-48aa-9b00-a9523ace4547" (UID: "20d3e8bd-7666-48aa-9b00-a9523ace4547"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 15:03:00.351273 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.351254    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d3e8bd-7666-48aa-9b00-a9523ace4547-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20d3e8bd-7666-48aa-9b00-a9523ace4547" (UID: "20d3e8bd-7666-48aa-9b00-a9523ace4547"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 15:03:00.449172 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.449073    2565 generic.go:356] "Generic (PLEG): container finished" podID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerID="70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75" exitCode=0
Jul 07 15:03:00.449172 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.449114    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"
Jul 07 15:03:00.449172 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.449160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" event={"ID":"20d3e8bd-7666-48aa-9b00-a9523ace4547","Type":"ContainerDied","Data":"70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75"}
Jul 07 15:03:00.449384 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.449216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt" event={"ID":"20d3e8bd-7666-48aa-9b00-a9523ace4547","Type":"ContainerDied","Data":"9fb2246c7aebfe363fe75c18c17ea47a68d1214a1c275774c1307992fc63837b"}
Jul 07 15:03:00.449384 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.449243    2565 scope.go:117] "RemoveContainer" containerID="70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75"
Jul 07 15:03:00.449814 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.449796    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20d3e8bd-7666-48aa-9b00-a9523ace4547-proxy-tls\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 15:03:00.449909 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.449816    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20d3e8bd-7666-48aa-9b00-a9523ace4547-openshift-service-ca-bundle\") on node \"ip-10-0-138-82.ec2.internal\" DevicePath \"\""
Jul 07 15:03:00.458323 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.458303    2565 scope.go:117] "RemoveContainer" containerID="70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75"
Jul 07 15:03:00.458676 ip-10-0-138-82 kubenswrapper[2565]: E0707 15:03:00.458655    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75\": container with ID starting with 70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75 not found: ID does not exist" containerID="70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75"
Jul 07 15:03:00.458737 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.458684    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75"} err="failed to get container status \"70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75\": rpc error: code = NotFound desc = could not find container \"70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75\": container with ID starting with 70db584d94b1b89c3ae8a365de6b36a4db1110f2ccaf2637f1f0448bc6481b75 not found: ID does not exist"
Jul 07 15:03:00.479552 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.479519    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"]
Jul 07 15:03:00.487293 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:00.487266    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-36f00-7bc7bf4797-g5dbt"]
Jul 07 15:03:01.485724 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:01.485612    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" path="/var/lib/kubelet/pods/20d3e8bd-7666-48aa-9b00-a9523ace4547/volumes"
Jul 07 15:03:06.510357 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:06.510328    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m6ghz_e8ae7e96-9ae1-45b4-a74a-fb810990dd5c/global-pull-secret-syncer/0.log"
Jul 07 15:03:06.676361 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:06.676326    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p66vc_b0b57686-10e6-4348-b196-2a49ace4083f/konnectivity-agent/0.log"
Jul 07 15:03:06.757714 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:06.757682    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-82.ec2.internal_50cab86694ee5c9634e52b1e02eef957/haproxy/0.log"
Jul 07 15:03:11.188558 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:11.188528    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-8hgbl_4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10/kube-state-metrics/0.log"
Jul 07 15:03:11.216929 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:11.216902    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-8hgbl_4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10/kube-rbac-proxy-main/0.log"
Jul 07 15:03:11.240408 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:11.240374    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-8hgbl_4b7cdcd9-aab7-4abd-9a93-ca0f6d76ae10/kube-rbac-proxy-self/0.log"
Jul 07 15:03:11.358766 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:11.358729    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77z5d_4f9569e9-6251-4009-aabf-9b91b7b459af/node-exporter/0.log"
Jul 07 15:03:11.387743 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:11.387719    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77z5d_4f9569e9-6251-4009-aabf-9b91b7b459af/kube-rbac-proxy/0.log"
Jul 07 15:03:11.415647 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:11.415596    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77z5d_4f9569e9-6251-4009-aabf-9b91b7b459af/init-textfile/0.log"
Jul 07 15:03:11.661156 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:11.661129    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-ms8qd_81b8fcc8-7f7d-473f-9218-d472e83f1835/kube-rbac-proxy-main/0.log"
Jul 07 15:03:11.701446 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:11.701414    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-ms8qd_81b8fcc8-7f7d-473f-9218-d472e83f1835/kube-rbac-proxy-self/0.log"
Jul 07 15:03:11.728933 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:11.728903    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-ms8qd_81b8fcc8-7f7d-473f-9218-d472e83f1835/openshift-state-metrics/0.log"
Jul 07 15:03:15.475870 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.475836    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"]
Jul 07 15:03:15.476636 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476594    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kube-rbac-proxy"
Jul 07 15:03:15.476636 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476636    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kube-rbac-proxy"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476661    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kube-rbac-proxy"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476670    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kube-rbac-proxy"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476680    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476688    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476703    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476711    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476719    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476727    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476743    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container"
Jul 07 15:03:15.476803 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476752    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container"
Jul 07 15:03:15.477288 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476824    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kserve-container"
Jul 07 15:03:15.477288 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476836    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d3e8bd-7666-48aa-9b00-a9523ace4547" containerName="switch-graph-36f00"
Jul 07 15:03:15.477288 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476847    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c525a2c-686b-4c1c-afd0-e826d769d80a" containerName="kube-rbac-proxy"
Jul 07 15:03:15.477288 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476856    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kube-rbac-proxy"
Jul 07 15:03:15.477288 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476865    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="245eaf0e-81d9-4381-a501-33bb7f9952ef" containerName="kserve-container"
Jul 07 15:03:15.477288 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.476875    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4d40633-686a-4b6b-a8fa-b4cabeff4f57" containerName="splitter-graph-7beb0"
Jul 07 15:03:15.481585 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.481560    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.486254 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.486233    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nrhtp\"/\"openshift-service-ca.crt\""
Jul 07 15:03:15.487053 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.487035    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nrhtp\"/\"kube-root-ca.crt\""
Jul 07 15:03:15.487151 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.487043    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nrhtp\"/\"default-dockercfg-w6fgr\""
Jul 07 15:03:15.489549 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.489528    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"]
Jul 07 15:03:15.557665 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.557602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-proc\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.557849 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.557685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-podres\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.557849 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.557719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvsv\" (UniqueName: \"kubernetes.io/projected/306e1495-33af-4cdf-b03d-0403e23d8fec-kube-api-access-2pvsv\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.557849 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.557740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-lib-modules\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.557849 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.557764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-sys\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.658735 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.658695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-proc\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.658925 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.658751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-podres\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.658925 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.658782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvsv\" (UniqueName: \"kubernetes.io/projected/306e1495-33af-4cdf-b03d-0403e23d8fec-kube-api-access-2pvsv\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.658925 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.658800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-lib-modules\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.658925 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.658824    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-proc\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.658925 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.658826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-sys\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.658925 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.658878    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-sys\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.659226 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.658950    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-podres\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.659226 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.658966    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/306e1495-33af-4cdf-b03d-0403e23d8fec-lib-modules\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.671007 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.670976    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvsv\" (UniqueName: \"kubernetes.io/projected/306e1495-33af-4cdf-b03d-0403e23d8fec-kube-api-access-2pvsv\") pod \"perf-node-gather-daemonset-r825c\" (UID: \"306e1495-33af-4cdf-b03d-0403e23d8fec\") " pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.791696 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.791567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:15.924601 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:15.924564    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"]
Jul 07 15:03:15.927646 ip-10-0-138-82 kubenswrapper[2565]: W0707 15:03:15.927596    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod306e1495_33af_4cdf_b03d_0403e23d8fec.slice/crio-85e74108760c44607bbfdc2ae6abb036d051d5ce856c264d61f62d0d2a1c53b2 WatchSource:0}: Error finding container 85e74108760c44607bbfdc2ae6abb036d051d5ce856c264d61f62d0d2a1c53b2: Status 404 returned error can't find the container with id 85e74108760c44607bbfdc2ae6abb036d051d5ce856c264d61f62d0d2a1c53b2
Jul 07 15:03:16.223362 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:16.223328    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bbxsw_7bc20856-2dc3-409d-aa1c-92eb8ff59349/dns/0.log"
Jul 07 15:03:16.245927 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:16.245891    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bbxsw_7bc20856-2dc3-409d-aa1c-92eb8ff59349/kube-rbac-proxy/0.log"
Jul 07 15:03:16.383983 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:16.383951    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gj9qr_d185cfb0-b3b3-4ef3-acbe-ea86e5ad0fed/dns-node-resolver/0.log"
Jul 07 15:03:16.509635 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:16.509528    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c" event={"ID":"306e1495-33af-4cdf-b03d-0403e23d8fec","Type":"ContainerStarted","Data":"55cc8e588194da1ce4c96a236fcac446bd87859997532860fedf3dcf2cb5c9e4"}
Jul 07 15:03:16.509635 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:16.509579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c" event={"ID":"306e1495-33af-4cdf-b03d-0403e23d8fec","Type":"ContainerStarted","Data":"85e74108760c44607bbfdc2ae6abb036d051d5ce856c264d61f62d0d2a1c53b2"}
Jul 07 15:03:16.510020 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:16.509653    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:16.526334 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:16.526257    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c" podStartSLOduration=1.526239892 podStartE2EDuration="1.526239892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:03:16.525152006 +0000 UTC m=+3247.522742731" watchObservedRunningTime="2026-07-07 15:03:16.526239892 +0000 UTC m=+3247.523830649"
Jul 07 15:03:16.886718 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:16.886681    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-679d4cb8b5-mjvw7_fd4b52a9-eef8-4ad9-bd0a-1ba6efc91927/registry/0.log"
Jul 07 15:03:16.962963 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:16.962930    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9xdbg_fd47c6c1-367e-416b-8c25-b64c869d5c22/node-ca/0.log"
Jul 07 15:03:18.513545 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:18.513513    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wkpv6_c200e335-efdd-4d8b-bb88-42fa4bfecf13/serve-healthcheck-canary/0.log"
Jul 07 15:03:19.122578 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:19.122533    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dq65l_e9b9bfcb-3ffe-4b06-bdea-a0be6428424f/kube-rbac-proxy/0.log"
Jul 07 15:03:19.153164 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:19.153135    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dq65l_e9b9bfcb-3ffe-4b06-bdea-a0be6428424f/exporter/0.log"
Jul 07 15:03:19.190114 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:19.190086    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dq65l_e9b9bfcb-3ffe-4b06-bdea-a0be6428424f/extractor/0.log"
Jul 07 15:03:21.604471 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:21.604439    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-xs5z2_a8bb4e7b-2dd5-4415-9612-b590440c162d/manager/0.log"
Jul 07 15:03:21.780526 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:21.780489    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-jzz99_06bcbd16-2cac-4ecb-b017-12d1f26f7b14/manager/0.log"
Jul 07 15:03:21.950400 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:21.950365    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qmqvs_771a59f7-d5c9-4b4d-a2f4-e04e6bd251ca/seaweedfs/0.log"
Jul 07 15:03:22.522416 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:22.522389    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nrhtp/perf-node-gather-daemonset-r825c"
Jul 07 15:03:27.756778 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:27.756749    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54wn6_c78112d6-bdd7-44c5-9214-04e37f5c16a2/kube-multus-additional-cni-plugins/0.log"
Jul 07 15:03:27.803203 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:27.803173    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54wn6_c78112d6-bdd7-44c5-9214-04e37f5c16a2/egress-router-binary-copy/0.log"
Jul 07 15:03:27.825539 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:27.825513    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54wn6_c78112d6-bdd7-44c5-9214-04e37f5c16a2/cni-plugins/0.log"
Jul 07 15:03:27.847165 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:27.847138    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54wn6_c78112d6-bdd7-44c5-9214-04e37f5c16a2/bond-cni-plugin/0.log"
Jul 07 15:03:27.870802 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:27.870764    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54wn6_c78112d6-bdd7-44c5-9214-04e37f5c16a2/routeoverride-cni/0.log"
Jul 07 15:03:27.895293 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:27.895262    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54wn6_c78112d6-bdd7-44c5-9214-04e37f5c16a2/whereabouts-cni-bincopy/0.log"
Jul 07 15:03:27.917727 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:27.917698    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54wn6_c78112d6-bdd7-44c5-9214-04e37f5c16a2/whereabouts-cni/0.log"
Jul 07 15:03:28.369693 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:28.369659    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xtl8q_46a39168-6bed-493d-a1e0-c4313508adc4/kube-multus/0.log"
Jul 07 15:03:28.500056 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:28.500024    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-znqh7_a92bcc45-e37c-46ed-9887-6a895f7e7104/network-metrics-daemon/0.log"
Jul 07 15:03:28.534563 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:28.534527    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-znqh7_a92bcc45-e37c-46ed-9887-6a895f7e7104/kube-rbac-proxy/0.log"
Jul 07 15:03:29.316386 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:29.316358    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82xpw_7b291786-af74-43e3-84e4-2af9581a134f/ovn-controller/0.log"
Jul 07 15:03:29.335950 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:29.335922    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82xpw_7b291786-af74-43e3-84e4-2af9581a134f/ovn-acl-logging/0.log"
Jul 07 15:03:29.367697 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:29.367663    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82xpw_7b291786-af74-43e3-84e4-2af9581a134f/kube-rbac-proxy-node/0.log"
Jul 07 15:03:29.397302 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:29.397269    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82xpw_7b291786-af74-43e3-84e4-2af9581a134f/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 15:03:29.417118 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:29.417085    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82xpw_7b291786-af74-43e3-84e4-2af9581a134f/northd/0.log"
Jul 07 15:03:29.442352 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:29.442323    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82xpw_7b291786-af74-43e3-84e4-2af9581a134f/nbdb/0.log"
Jul 07 15:03:29.466356 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:29.466332    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82xpw_7b291786-af74-43e3-84e4-2af9581a134f/sbdb/0.log"
Jul 07 15:03:29.532062 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:29.532031    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-82xpw_7b291786-af74-43e3-84e4-2af9581a134f/ovnkube-controller/0.log"
Jul 07 15:03:31.003728 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:31.003695    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dh95c_e9fd18af-03b0-434f-9ef3-a9b2775d545f/network-check-target-container/0.log"
Jul 07 15:03:31.991746 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:31.991721    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bhjlt_4ee39c5e-114a-4756-b5e3-f68d0d72c49b/iptables-alerter/0.log"
Jul 07 15:03:32.675506 ip-10-0-138-82 kubenswrapper[2565]: I0707 15:03:32.675478    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-kdp5h_177c9d3c-7f21-498a-b90d-1c818b61a216/tuned/0.log"