Jul 24 21:13:13.141203 ip-10-0-129-163 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 24 21:13:13.141214 ip-10-0-129-163 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 24 21:13:13.141222 ip-10-0-129-163 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 24 21:13:13.141547 ip-10-0-129-163 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 24 21:13:24.457741 ip-10-0-129-163 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 24 21:13:24.457759 ip-10-0-129-163 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 61cc873f52d94882bce7ade9b725bb50 --
Jul 24 21:15:25.536714 ip-10-0-129-163 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 21:15:25.972509 ip-10-0-129-163 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:15:25.972509 ip-10-0-129-163 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 21:15:25.972509 ip-10-0-129-163 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:15:25.972509 ip-10-0-129-163 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 21:15:25.972509 ip-10-0-129-163 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:15:25.974250 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.974156    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 21:15:25.978912 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978895    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:25.978912 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978913    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978917    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978921    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978924    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978927    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978930    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978933    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978936    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978938    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978945    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978948    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978951    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978954    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978957    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978960    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978962    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978965    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978968    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978970    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978973    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:25.978981 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978976    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978979    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978982    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978985    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978987    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978990    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978993    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978996    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.978999    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979001    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979004    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979006    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979009    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979011    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979014    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979017    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979019    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979022    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979026    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:25.979479 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979030    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979033    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979036    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979039    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979042    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979045    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979048    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979051    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979054    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979056    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979060    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979063    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979066    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979069    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979072    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979075    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979078    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979081    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979084    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:25.979937 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979087    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979092    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979096    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979100    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979102    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979105    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979108    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979110    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979114    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979117    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979119    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979123    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979126    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979129    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979132    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979134    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979137    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979140    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979144    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979147    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:25.980421 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979149    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979152    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979154    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979157    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979160    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979162    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979165    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979168    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979172    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979175    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979177    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979180    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979184    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979188    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979191    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979194    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979197    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979200    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979202    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.979206    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:25.980906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980719    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980727    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980731    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980734    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980737    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980740    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980744    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980747    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980749    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980753    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980755    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980758    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980760    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980763    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980766    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980769    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980772    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980774    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980777    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980780    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:25.981401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980783    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980785    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980788    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980792    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980794    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980798    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980800    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980803    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980805    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980808    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980810    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980813    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980815    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980818    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980822    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980826    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980829    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980831    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980835    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:25.981906 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980838    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980841    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980843    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980845    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980848    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980850    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980854    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980858    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980861    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980864    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980867    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980870    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980873    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980876    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980878    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980881    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980883    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980886    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980888    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:25.982381 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980891    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980894    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980896    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980899    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980902    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980904    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980907    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980909    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980912    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980915    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980918    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980920    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980924    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980927    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980930    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980932    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980935    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980938    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980940    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980943    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980945    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:25.982895 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980949    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980951    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980954    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980957    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980960    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980964    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980968    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980971    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980974    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980977    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980979    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980982    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980985    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980988    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980991    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980993    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980996    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.980999    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981002    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981004    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:25.983420 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981464    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981471    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981474    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981478    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981482    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981499    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981503    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981506    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981509    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981513    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981516    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981519    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981521    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981524    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981527    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981530    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981533    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981536    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981538    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981542    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:25.983914 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981545    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981547    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981550    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981553    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981556    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981558    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981561    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981564    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981568    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981571    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981574    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981577    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981579    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981582    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981585    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981589    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981592    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981596    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981598    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:25.984441 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981601    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981604    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981606    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981609    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981612    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981614    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981617    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981620    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981622    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981625    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981627    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981630    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981632    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981635    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981637    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981640    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981642    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981645    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981647    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:25.984903 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981650    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981654    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981657    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981660    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981663    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981666    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981668    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981671    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981674    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981677    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981679    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981682    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981685    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981687    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981690    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981692    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981695    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981698    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981701    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:25.985357 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981703    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981707    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981711    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981713    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981716    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981719    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981722    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981724    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981727    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981729    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981732    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981735    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981738    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981740    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981743    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981746    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981748    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981751    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981753    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:25.985856 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981756    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981761    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.981764    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981832    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981840    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981855    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981862    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981869    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981874    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981881    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981886    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981890    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981893    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981897    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981900    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981903    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981907    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981910    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981913    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981916    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981919    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981923    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981926    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981929    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 21:15:25.986325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981934    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981938    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981941    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981945    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981948    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981951    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981954    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981957    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981960    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981965    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981968    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981972    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981976    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981979    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981982    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981984    2580 flags.go:64] FLAG: --enable-server="true"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981988    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.981993    2580 flags.go:64] FLAG: --event-burst="100"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982005    2580 flags.go:64] FLAG: --event-qps="50"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982008    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982012    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982014    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982018    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982021    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982024    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 21:15:25.986912 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982027    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982030    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982033    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982036    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982039    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982042    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982045    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982049    2580 flags.go:64] FLAG: --feature-gates=""
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982053    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982056    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982059    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982062    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982066    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982069    2580 flags.go:64] FLAG: --help="false"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982072    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-129-163.ec2.internal"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982075    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982079    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982082    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982086    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982089    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982092    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982095    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982098    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 21:15:25.987527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982101    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982104    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982108    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982111    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982113    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982116    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982119    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982122    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982125    2580 flags.go:64] FLAG: --lock-file=""
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982128    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982131    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982134    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982139    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982142    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982145    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982148    2580 flags.go:64] FLAG: --logging-format="text"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982153    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982156    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982159    2580 flags.go:64] FLAG: --manifest-url=""
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982162    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982166    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982169    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982173    2580 flags.go:64] FLAG: --max-pods="110"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982176    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982186    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 21:15:25.988120 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982190    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982193    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982196    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982199    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982202    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982210    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982213    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982216    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982219    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982223    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982228    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982231    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982235    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982237    2580 flags.go:64] FLAG: --port="10250"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982241    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982244    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0483bacff1c411f4b"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982247    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982250    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982253    2580 flags.go:64] FLAG: --register-node="true"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982256    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982260    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982262    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982265    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982269    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982274    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 21:15:25.988790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982277    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982280    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982283    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982286    2580 flags.go:64] FLAG: --runonce="false"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982289    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982292    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982295    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982299    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982303    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982306    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982309    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982312    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982315    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982318    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982321    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982323    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982326    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982330    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982333    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982339    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982342    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982345    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982349    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982351    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982354    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 21:15:25.989445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982357    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982360    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982363    2580 flags.go:64] FLAG: --v="2"
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982367    2580 flags.go:64] FLAG: --version="false"
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982372    2580 flags.go:64] FLAG: --vmodule=""
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982377    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.982380    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982509    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982513    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982517    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982521    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982525    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982528    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982531    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982535    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982538    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982541    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982543    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982546    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982549    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982551    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:25.990050 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982554    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982557    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982560    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982562    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982566    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982568    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982571    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982574    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982576    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982579    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982581    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982584    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982587    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982589    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982592    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982594    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982598    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982601    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982603    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:25.990598 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982606    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982608    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982611    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982614    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982616    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982619    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982621    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982624    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982627    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982629    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982631    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982634    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982636    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982639    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982642    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982644    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982647    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982649    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982653    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982655    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:25.991057 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982658    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982660    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982663    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982665    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982668    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982670    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982673    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982675    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982678    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982682    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982684    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982687    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982689    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982692    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982694    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982697    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982700    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982702    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982705    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982707    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:25.991659 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982711    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982714    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982718    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982721    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982724    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982726    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982729    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982731    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982734    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982736    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982740    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982744    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982747    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982750    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982753    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982756    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982759    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982761    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982764    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:25.992157 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982766    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982769    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982773    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982775    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982778    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982780    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.982783    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.983495    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.990148    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.990166    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.990199    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.990206    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990270    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990275    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990279    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:25.992643 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990282    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990285    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990288    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990291    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990293    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990296    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990299    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990302    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990304    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990307    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990310    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990312    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990315    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990317    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990320    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990322    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990325    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990329    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990332    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990335    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:25.993025 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990337    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990340    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990343    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990345    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990348    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990350    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990356    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990359    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990362    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990365    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990368    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990370    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990373    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990376    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990378    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990381    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990384    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990401    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990405    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:25.993544 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990408    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990411    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990414    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990417    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990420    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990423    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990426    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990428    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990431    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990434    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990437    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990439    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990442    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990445    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990448    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990452    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990455    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990458    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990461    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:25.994003 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990465    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990469    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990471    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990474    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990477    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990479    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990482    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990484    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990487    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990489    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990492    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990494    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990497    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990499    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990502    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990504    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990507    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990509    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990512    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990514    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:25.994503 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990517    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990519    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990522    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990524    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990527    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990530    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990533    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990536    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990538    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990541    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990543    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990546    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990549    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990551    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990554    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990557    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990559    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:25.994991 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990562    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.990567    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990709    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990715    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990718    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990721    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990723    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990726    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990729    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990731    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990734    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990737    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990740    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990742    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990745    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:25.995401 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990747    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990750    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990753    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990755    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990757    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990760    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990763    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990765    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990768    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990770    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990773    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990776    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990779    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990781    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990784    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990786    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990789    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990792    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990794    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990796    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:25.995786 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990799    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990802    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990804    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990807    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990809    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990812    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990814    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990817    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990820    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990824    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990827    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990830    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990833    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990836    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990839    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990842    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990844    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990847    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990850    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990852    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:25.996270 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990855    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990857    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990860    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990863    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990865    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990869    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990872    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990875    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990879    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990881    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990884    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990888    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990891    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990894    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990896    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990900    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990902    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990905    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990908    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:25.996773 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990911    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990914    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990916    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990919    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990922    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990925    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990927    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990930    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990933    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990935    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990938    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990940    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990943    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990945    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990948    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990950    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990953    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990955    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990958    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:25.997244 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990960    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990963    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990966    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990968    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990971    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990973    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990976    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:25.990978    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.990983    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.991118    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.993258    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.994462    2580 server.go:1019] "Starting client certificate rotation"
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.994574    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 21:15:25.997910 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.994616    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 21:15:25.998411 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:25.996560    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 21:15:26.002988 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.002966    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 21:15:26.012789 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.012770    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 24 21:15:26.014780 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.014764    2580 log.go:25] "Validated CRI v1 image API"
Jul 24 21:15:26.015700 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.015679    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 21:15:26.021358 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.021338    2580 fs.go:135] Filesystem UUIDs: map[65856127-e8ef-47b0-af4e-088151cce6af:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dccf436b-b9fe-44e5-8120-5a8a4e2e060c:/dev/nvme0n1p4]
Jul 24 21:15:26.021466 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.021358    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 21:15:26.026474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.026332    2580 manager.go:216] Machine: {Timestamp:2026-07-24 21:15:26.025189453 +0000 UTC m=+0.364031220 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2957792 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2357be36e4f5d7f7f75b1edc800055 SystemUUID:ec2357be-36e4-f5d7-f7f7-5b1edc800055 BootID:61cc873f-52d9-4882-bce7-ade9b725bb50 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e3:72:93:47:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e3:72:93:47:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:e8:32:f8:9d:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 21:15:26.026474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.026463    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 21:15:26.026588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.026559    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 21:15:26.027607 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.027580    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 21:15:26.027785 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.027612    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 21:15:26.028477 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.028463    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 21:15:26.028540 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.028482    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 21:15:26.028540 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.028494    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 21:15:26.029920 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.029907    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 21:15:26.030703 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.030690    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 21:15:26.031940 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.031921    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 21:15:26.032603 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.032589    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 21:15:26.032747 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.032735    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 21:15:26.032801 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.032777    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 21:15:26.032848 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.032798    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 21:15:26.032848 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.032817    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 21:15:26.032848 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.032830    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 24 21:15:26.032848 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.032843    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 21:15:26.033229 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.033215    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 21:15:26.033290 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.033241    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 21:15:26.036441 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.036423    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 21:15:26.040526 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.040508    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 21:15:26.040589 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.040541    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 21:15:26.041915 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041900    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 21:15:26.041971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041924    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 21:15:26.041971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041933    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 21:15:26.041971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041939    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 21:15:26.041971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041948    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 21:15:26.041971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041954    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 21:15:26.041971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041962    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 21:15:26.041971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041967    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 21:15:26.042175 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041979    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 21:15:26.042175 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.041988    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 21:15:26.042175 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.042000    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 21:15:26.042175 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.042006    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 21:15:26.042175 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.042029    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 21:15:26.042175 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.042040    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 21:15:26.043413 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.043403    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 21:15:26.043413 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.043413    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 21:15:26.046438 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.046423    2580 server.go:1268] "Started kubelet"
Jul 24 21:15:26.046527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.046502    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 21:15:26.046744 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.046654    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 21:15:26.046847 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.046771    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 21:15:26.047330 ip-10-0-129-163 systemd[1]: Started Kubernetes Kubelet.
Jul 24 21:15:26.048445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.048431    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 21:15:26.050588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.050569    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 21:15:26.050680 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.050612    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 21:15:26.053503 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.053487    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t8d76"
Jul 24 21:15:26.054013 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.053998    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 21:15:26.054946 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.054050    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-163.ec2.internal.18c5564f54bd8071  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-163.ec2.internal,UID:ip-10-0-129-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-163.ec2.internal,},FirstTimestamp:2026-07-24 21:15:26.046380145 +0000 UTC m=+0.385221912,LastTimestamp:2026-07-24 21:15:26.046380145 +0000 UTC m=+0.385221912,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-163.ec2.internal,}"
Jul 24 21:15:26.056179 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.056164    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 21:15:26.056228 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.056179    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 21:15:26.056837 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.056320    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.056837 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.056662    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 21:15:26.056837 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.056703    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 21:15:26.056837 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.056761    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 21:15:26.057230 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.057105    2580 factory.go:55] Registering systemd factory
Jul 24 21:15:26.057230 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.057126    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 24 21:15:26.057230 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.057195    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 21:15:26.057230 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.057205    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 21:15:26.057459 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.057443    2580 factory.go:153] Registering CRI-O factory
Jul 24 21:15:26.057459 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.057457    2580 factory.go:223] Registration of the crio container factory successfully
Jul 24 21:15:26.057566 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.057551    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 21:15:26.057607 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.057588    2580 factory.go:103] Registering Raw factory
Jul 24 21:15:26.057646 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.057608    2580 manager.go:1196] Started watching for new ooms in manager
Jul 24 21:15:26.058312 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.058239    2580 manager.go:349] Starting recovery of all containers
Jul 24 21:15:26.059403 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.059354    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 21:15:26.059553 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.059527    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 21:15:26.061137 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.061111    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t8d76"
Jul 24 21:15:26.070268 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.070095    2580 manager.go:354] Recovery completed
Jul 24 21:15:26.075298 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.075282    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:26.077229 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.077212    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:26.077303 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.077246    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:26.077303 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.077259    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:26.077831 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.077818    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 21:15:26.077831 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.077828    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 21:15:26.077938 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.077834    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:26.077938 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.077842    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 21:15:26.078597 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.078574    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:26.078672 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.078600    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:26.078672 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.078613    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:26.079372 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.079306    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-163.ec2.internal.18c5564f56943e4f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-163.ec2.internal,UID:ip-10-0-129-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-163.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-163.ec2.internal,},FirstTimestamp:2026-07-24 21:15:26.077230671 +0000 UTC m=+0.416072441,LastTimestamp:2026-07-24 21:15:26.077230671 +0000 UTC m=+0.416072441,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-163.ec2.internal,}"
Jul 24 21:15:26.079514 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.079501    2580 policy_none.go:49] "None policy: Start"
Jul 24 21:15:26.079545 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.079523    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 21:15:26.079545 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.079538    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 21:15:26.080517 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.080506    2580 policy_none.go:47] "Start"
Jul 24 21:15:26.117004 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.116978    2580 manager.go:337] "Starting Device Plugin manager"
Jul 24 21:15:26.117142 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.117023    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 21:15:26.117142 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.117036    2580 server.go:87] "Starting device plugin registration server"
Jul 24 21:15:26.117350 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.117336    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 21:15:26.117402 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.117355    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 21:15:26.117519 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.117500    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 21:15:26.117598 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.117583    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 21:15:26.117598 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.117595    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 21:15:26.118134 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.118115    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 21:15:26.118212 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.118161    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.217657 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.217618    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:26.218862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.218606    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:26.218862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.218634    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:26.218862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.218671    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:26.218862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.218712    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.227195 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.227148    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.227195 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.227172    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-163.ec2.internal\": node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.243482 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.243459    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.279626 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.279588    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 21:15:26.281010 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.280991    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 21:15:26.281102 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.281034    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 21:15:26.281102 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.281055    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 21:15:26.281102 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.281093    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 21:15:26.283357 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.283335    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:26.344370 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.344321    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.381786 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.381753    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"]
Jul 24 21:15:26.381853 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.381828    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:26.382922 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.382899    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:26.383016 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.382939    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:26.383016 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.382951    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:26.384331 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.384318    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:26.384465 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.384452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.384504 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.384481    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:26.385160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.385138    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:26.385264 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.385171    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:26.385264 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.385173    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:26.385264 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.385185    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:26.385264 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.385198    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:26.385264 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.385208    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:26.386839 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.386826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.386904 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.386850    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:26.387571 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.387555    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:26.387652 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.387585    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:26.387652 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.387598    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:26.422201 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.422170    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-163.ec2.internal\" not found" node="ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.425708 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.425690    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-163.ec2.internal\" not found" node="ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.445323 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.445295    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.459421 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.459370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d9789b24e4fc6e162649ac96a8871367-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"d9789b24e4fc6e162649ac96a8871367\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.459421 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.459419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9789b24e4fc6e162649ac96a8871367-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"d9789b24e4fc6e162649ac96a8871367\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.459606 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.459438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a258609098d82a70d5e5404dc8cfce27-config\") pod \"kube-apiserver-proxy-ip-10-0-129-163.ec2.internal\" (UID: \"a258609098d82a70d5e5404dc8cfce27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.546354 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.546266    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.559577 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.559545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9789b24e4fc6e162649ac96a8871367-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"d9789b24e4fc6e162649ac96a8871367\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.559667 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.559584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a258609098d82a70d5e5404dc8cfce27-config\") pod \"kube-apiserver-proxy-ip-10-0-129-163.ec2.internal\" (UID: \"a258609098d82a70d5e5404dc8cfce27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.559667 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.559609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d9789b24e4fc6e162649ac96a8871367-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"d9789b24e4fc6e162649ac96a8871367\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.559767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.559658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9789b24e4fc6e162649ac96a8871367-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"d9789b24e4fc6e162649ac96a8871367\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.559767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.559719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a258609098d82a70d5e5404dc8cfce27-config\") pod \"kube-apiserver-proxy-ip-10-0-129-163.ec2.internal\" (UID: \"a258609098d82a70d5e5404dc8cfce27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.559767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.559759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d9789b24e4fc6e162649ac96a8871367-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal\" (UID: \"d9789b24e4fc6e162649ac96a8871367\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.646770 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.646724    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.725260 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.725220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.730001 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.729973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:26.746968 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.746948    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.848048 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.848023    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.948439 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:26.948404    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:26.993788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.993755    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 21:15:26.994244 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:26.993903    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 21:15:27.049223 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:27.049181    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-163.ec2.internal\" not found"
Jul 24 21:15:27.065936 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.065899    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 21:10:25 +0000 UTC" deadline="2028-01-20 12:28:28.595072487 +0000 UTC"
Jul 24 21:15:27.065936 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.065933    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13071h13m1.529142851s"
Jul 24 21:15:27.133650 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.133630    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:27.156603 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.156576    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:27.166916 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.166895    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 21:15:27.168824 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.168811    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal"
Jul 24 21:15:27.174856 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.174830    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 21:15:27.223691 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:27.223659    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9789b24e4fc6e162649ac96a8871367.slice/crio-cb255b0be12e9121c63cd6bb57127b972d28011eaf12cabfd9c92a8eb90a6667 WatchSource:0}: Error finding container cb255b0be12e9121c63cd6bb57127b972d28011eaf12cabfd9c92a8eb90a6667: Status 404 returned error can't find the container with id cb255b0be12e9121c63cd6bb57127b972d28011eaf12cabfd9c92a8eb90a6667
Jul 24 21:15:27.223884 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:27.223868    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda258609098d82a70d5e5404dc8cfce27.slice/crio-2111a76b0d8988a2354a8d1afbd9a6a2a29db4cfbeaa85ab484502debb634d63 WatchSource:0}: Error finding container 2111a76b0d8988a2354a8d1afbd9a6a2a29db4cfbeaa85ab484502debb634d63: Status 404 returned error can't find the container with id 2111a76b0d8988a2354a8d1afbd9a6a2a29db4cfbeaa85ab484502debb634d63
Jul 24 21:15:27.229324 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.229306    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:15:27.284756 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.284696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" event={"ID":"d9789b24e4fc6e162649ac96a8871367","Type":"ContainerStarted","Data":"cb255b0be12e9121c63cd6bb57127b972d28011eaf12cabfd9c92a8eb90a6667"}
Jul 24 21:15:27.285665 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.285646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal" event={"ID":"a258609098d82a70d5e5404dc8cfce27","Type":"ContainerStarted","Data":"2111a76b0d8988a2354a8d1afbd9a6a2a29db4cfbeaa85ab484502debb634d63"}
Jul 24 21:15:27.361415 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.361372    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:27.645265 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:27.645233    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:28.033786 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.033706    2580 apiserver.go:52] "Watching apiserver"
Jul 24 21:15:28.040973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.040951    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 21:15:28.041762 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.041407    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-q6tdh","openshift-multus/network-metrics-daemon-njtn4","openshift-ovn-kubernetes/ovnkube-node-d5rxk","kube-system/global-pull-secret-syncer-xc9pb","kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb","openshift-image-registry/node-ca-qtpfl","openshift-network-diagnostics/network-check-target-8vg5v","openshift-network-operator/iptables-alerter-pbjmp","kube-system/konnectivity-agent-7v2tr","openshift-cluster-node-tuning-operator/tuned-cjvwv","openshift-dns/node-resolver-zvggz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal","openshift-multus/multus-additional-cni-plugins-cgqmz"]
Jul 24 21:15:28.043508 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.043060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.044468 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.044179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:28.044468 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.044246    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:28.045745 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.045721    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c8kbp\""
Jul 24 21:15:28.046007 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.045989    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 21:15:28.046143 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.046130    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 21:15:28.046353 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.046339    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 21:15:28.046545 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.046527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.047722 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.047703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.047812 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.047767    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:28.048768 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.048745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 21:15:28.048934 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.048917    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 21:15:28.049336 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.049316    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bcjlp\""
Jul 24 21:15:28.049541 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.049526    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 21:15:28.049617 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.049559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.050209 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.050188    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 21:15:28.050292 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.050229    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 21:15:28.050292 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.050277    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 21:15:28.051062 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.051041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.051803 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.051784    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 21:15:28.052214 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.052133    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 21:15:28.052400 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.052367    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 21:15:28.052603 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.052584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 21:15:28.052987 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.052910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.053725 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.053707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.053901 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.053882    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8slhw\""
Jul 24 21:15:28.054748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.054308    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 21:15:28.054748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.054453    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h6fx9\""
Jul 24 21:15:28.054748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.054563    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 21:15:28.055103 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.055065    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 21:15:28.055323 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.055305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:28.058424 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.055457    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:28.058424 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.055866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkph7\""
Jul 24 21:15:28.058424 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.056356    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:15:28.058424 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.057317    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 21:15:28.058424 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.057557    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 21:15:28.058424 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.057810    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 21:15:28.058424 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.058013    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 21:15:28.058424 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.058274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xhm4w\""
Jul 24 21:15:28.058907 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.058582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:28.059680 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.059659    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 21:15:28.059756 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.059719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.060032 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.060014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.063896 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.062869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mqv6j\""
Jul 24 21:15:28.063896 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.063285    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r7jjw\""
Jul 24 21:15:28.063896 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.063431    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 21:15:28.063896 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.063620    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 21:15:28.064134 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.064072    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7shdc\""
Jul 24 21:15:28.064288 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.064272    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 21:15:28.064355 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.064339    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 21:15:28.064438 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.064422    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 21:15:28.064874 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.064855    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 21:15:28.065351 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.065192    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:15:28.066676 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-os-release\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.066788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.066788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-kubernetes\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.066788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066752    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-cnibin\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.066788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-run-netns\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.066788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-var-lib-kubelet\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.067025 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.067025 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-var-lib-kubelet\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.067025 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54-host\") pod \"node-ca-qtpfl\" (UID: \"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54\") " pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.067025 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnvh\" (UniqueName: \"kubernetes.io/projected/6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54-kube-api-access-ltnvh\") pod \"node-ca-qtpfl\" (UID: \"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54\") " pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.067025 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.066899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb587855-58e3-4504-83e8-71afb6f79cba-multus-daemon-config\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.067189 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-run-multus-certs\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.067224 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-sys-fs\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.067224 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14ca3ba3-0b3a-4063-9f59-e4f205927cfe-iptables-alerter-script\") pod \"iptables-alerter-pbjmp\" (UID: \"14ca3ba3-0b3a-4063-9f59-e4f205927cfe\") " pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.067282 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-slash\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.067331 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.067370 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-modprobe-d\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.067434 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-run\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.067434 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-registration-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.067496 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66v4z\" (UniqueName: \"kubernetes.io/projected/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-kube-api-access-66v4z\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.067528 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-cnibin\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.067528 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-tmp\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.067588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-socket-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.067588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-device-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.067671 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnhf\" (UniqueName: \"kubernetes.io/projected/514cfda0-541e-4b7d-91dd-dcb870c17692-kube-api-access-rqnhf\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:28.067671 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-log-socket\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.067732 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-run-ovn-kubernetes\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.067732 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff698d8b-a442-4540-9084-615541d3bcff-ovnkube-script-lib\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.067732 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-system-cni-dir\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.067819 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.067819 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.067819 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-run-k8s-cni-cncf-io\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.067926 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-multus-conf-dir\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.067926 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-etc-openvswitch\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.067926 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-run-openvswitch\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.067926 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-node-log\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.068051 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-cni-bin\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.068051 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.067965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff698d8b-a442-4540-9084-615541d3bcff-ovn-node-metrics-cert\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.068051 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.068149 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp2w7\" (UniqueName: \"kubernetes.io/projected/1328ac14-7eae-451f-af99-eb968a843e61-kube-api-access-lp2w7\") pod \"node-resolver-zvggz\" (UID: \"1328ac14-7eae-451f-af99-eb968a843e61\") " pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.068149 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-cni-netd\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.068149 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff698d8b-a442-4540-9084-615541d3bcff-env-overrides\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.068149 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9p9\" (UniqueName: \"kubernetes.io/projected/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-kube-api-access-dw9p9\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.068149 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-sysconfig\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.068288 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5h8m\" (UniqueName: \"kubernetes.io/projected/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-kube-api-access-n5h8m\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.068288 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1328ac14-7eae-451f-af99-eb968a843e61-tmp-dir\") pod \"node-resolver-zvggz\" (UID: \"1328ac14-7eae-451f-af99-eb968a843e61\") " pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.068288 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54-serviceca\") pod \"node-ca-qtpfl\" (UID: \"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54\") " pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.068288 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-tuned\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.068288 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-multus-cni-dir\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.068498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-os-release\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.068498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-var-lib-cni-multus\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.068498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-hostroot\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.068498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm5n5\" (UniqueName: \"kubernetes.io/projected/14ca3ba3-0b3a-4063-9f59-e4f205927cfe-kube-api-access-vm5n5\") pod \"iptables-alerter-pbjmp\" (UID: \"14ca3ba3-0b3a-4063-9f59-e4f205927cfe\") " pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.068498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-run-systemd\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.068498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-sysctl-conf\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.068770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-system-cni-dir\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.068770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.068770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-etc-kubernetes\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.068770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tng2b\" (UniqueName: \"kubernetes.io/projected/eb587855-58e3-4504-83e8-71afb6f79cba-kube-api-access-tng2b\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.068770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.068770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-systemd-units\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.068770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-systemd\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.068770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-lib-modules\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb587855-58e3-4504-83e8-71afb6f79cba-cni-binary-copy\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-kubelet\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-run-ovn\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff698d8b-a442-4540-9084-615541d3bcff-ovnkube-config\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6df404c6-1f92-408c-b16b-8697d4f86fdc-kubelet-config\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6df404c6-1f92-408c-b16b-8697d4f86fdc-dbus\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2-agent-certs\") pod \"konnectivity-agent-7v2tr\" (UID: \"4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2\") " pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.068971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14ca3ba3-0b3a-4063-9f59-e4f205927cfe-host-slash\") pod \"iptables-alerter-pbjmp\" (UID: \"14ca3ba3-0b3a-4063-9f59-e4f205927cfe\") " pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-sys\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-host\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvx4z\" (UniqueName: \"kubernetes.io/projected/ff698d8b-a442-4540-9084-615541d3bcff-kube-api-access-mvx4z\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2-konnectivity-ca\") pod \"konnectivity-agent-7v2tr\" (UID: \"4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2\") " pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-sysctl-d\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-var-lib-cni-bin\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.069115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-run-netns\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.069856 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1328ac14-7eae-451f-af99-eb968a843e61-hosts-file\") pod \"node-resolver-zvggz\" (UID: \"1328ac14-7eae-451f-af99-eb968a843e61\") " pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.069856 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-multus-socket-dir-parent\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.069856 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:28.069856 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.069225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-var-lib-openvswitch\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.071616 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.071594    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 21:15:28.103169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.103140    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rck89"
Jul 24 21:15:28.112403 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.112359    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rck89"
Jul 24 21:15:28.169476 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66v4z\" (UniqueName: \"kubernetes.io/projected/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-kube-api-access-66v4z\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.169644 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-cnibin\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.169644 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-tmp\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.169644 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-socket-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.169644 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-device-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.169644 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-cnibin\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.169644 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnhf\" (UniqueName: \"kubernetes.io/projected/514cfda0-541e-4b7d-91dd-dcb870c17692-kube-api-access-rqnhf\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:28.169644 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-log-socket\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-run-ovn-kubernetes\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-device-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff698d8b-a442-4540-9084-615541d3bcff-ovnkube-script-lib\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-system-cni-dir\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-log-socket\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-socket-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-run-ovn-kubernetes\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-system-cni-dir\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-run-k8s-cni-cncf-io\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.169977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.169964    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-multus-conf-dir\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-etc-openvswitch\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170084    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-multus-conf-dir\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-run-k8s-cni-cncf-io\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170110    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-etc-openvswitch\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-run-openvswitch\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-run-openvswitch\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-node-log\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-cni-bin\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-cni-bin\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170212    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-node-log\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff698d8b-a442-4540-9084-615541d3bcff-ovn-node-metrics-cert\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp2w7\" (UniqueName: \"kubernetes.io/projected/1328ac14-7eae-451f-af99-eb968a843e61-kube-api-access-lp2w7\") pod \"node-resolver-zvggz\" (UID: \"1328ac14-7eae-451f-af99-eb968a843e61\") " pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ff698d8b-a442-4540-9084-615541d3bcff-ovnkube-script-lib\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.170486 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-cni-netd\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff698d8b-a442-4540-9084-615541d3bcff-env-overrides\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-cni-netd\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.170460    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9p9\" (UniqueName: \"kubernetes.io/projected/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-kube-api-access-dw9p9\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.170529    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret podName:6df404c6-1f92-408c-b16b-8697d4f86fdc nodeName:}" failed. No retries permitted until 2026-07-24 21:15:28.670508666 +0000 UTC m=+3.009350426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret") pod "global-pull-secret-syncer-xc9pb" (UID: "6df404c6-1f92-408c-b16b-8697d4f86fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-sysconfig\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5h8m\" (UniqueName: \"kubernetes.io/projected/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-kube-api-access-n5h8m\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1328ac14-7eae-451f-af99-eb968a843e61-tmp-dir\") pod \"node-resolver-zvggz\" (UID: \"1328ac14-7eae-451f-af99-eb968a843e61\") " pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54-serviceca\") pod \"node-ca-qtpfl\" (UID: \"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54\") " pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-sysconfig\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-tuned\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-multus-cni-dir\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-os-release\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-var-lib-cni-multus\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-hostroot\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm5n5\" (UniqueName: \"kubernetes.io/projected/14ca3ba3-0b3a-4063-9f59-e4f205927cfe-kube-api-access-vm5n5\") pod \"iptables-alerter-pbjmp\" (UID: \"14ca3ba3-0b3a-4063-9f59-e4f205927cfe\") " pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.171335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-run-systemd\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ff698d8b-a442-4540-9084-615541d3bcff-env-overrides\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-var-lib-cni-multus\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1328ac14-7eae-451f-af99-eb968a843e61-tmp-dir\") pod \"node-resolver-zvggz\" (UID: \"1328ac14-7eae-451f-af99-eb968a843e61\") " pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-sysctl-conf\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-multus-cni-dir\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-hostroot\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.170841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-sysctl-conf\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-run-systemd\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-system-cni-dir\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54-serviceca\") pod \"node-ca-qtpfl\" (UID: \"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54\") " pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-os-release\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-etc-kubernetes\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-system-cni-dir\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tng2b\" (UniqueName: \"kubernetes.io/projected/eb587855-58e3-4504-83e8-71afb6f79cba-kube-api-access-tng2b\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-etc-kubernetes\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.172193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-systemd-units\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-systemd\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-lib-modules\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-systemd-units\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-systemd\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb587855-58e3-4504-83e8-71afb6f79cba-cni-binary-copy\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-kubelet\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-lib-modules\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-run-ovn\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-kubelet\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff698d8b-a442-4540-9084-615541d3bcff-ovnkube-config\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-run-ovn\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6df404c6-1f92-408c-b16b-8697d4f86fdc-kubelet-config\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6df404c6-1f92-408c-b16b-8697d4f86fdc-dbus\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2-agent-certs\") pod \"konnectivity-agent-7v2tr\" (UID: \"4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2\") " pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:28.173092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14ca3ba3-0b3a-4063-9f59-e4f205927cfe-host-slash\") pod \"iptables-alerter-pbjmp\" (UID: \"14ca3ba3-0b3a-4063-9f59-e4f205927cfe\") " pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6df404c6-1f92-408c-b16b-8697d4f86fdc-dbus\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-sys\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-host\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-sys\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvx4z\" (UniqueName: \"kubernetes.io/projected/ff698d8b-a442-4540-9084-615541d3bcff-kube-api-access-mvx4z\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2-konnectivity-ca\") pod \"konnectivity-agent-7v2tr\" (UID: \"4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2\") " pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-sysctl-d\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-var-lib-cni-bin\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-run-netns\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1328ac14-7eae-451f-af99-eb968a843e61-hosts-file\") pod \"node-resolver-zvggz\" (UID: \"1328ac14-7eae-451f-af99-eb968a843e61\") " pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-multus-socket-dir-parent\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-var-lib-openvswitch\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-os-release\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-kubernetes\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.173973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ff698d8b-a442-4540-9084-615541d3bcff-ovnkube-config\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.171973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-cnibin\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-run-netns\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-cnibin\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-var-lib-kubelet\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6df404c6-1f92-408c-b16b-8697d4f86fdc-kubelet-config\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-var-lib-kubelet\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-host\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54-host\") pod \"node-ca-qtpfl\" (UID: \"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54\") " pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnvh\" (UniqueName: \"kubernetes.io/projected/6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54-kube-api-access-ltnvh\") pod \"node-ca-qtpfl\" (UID: \"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54\") " pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb587855-58e3-4504-83e8-71afb6f79cba-multus-daemon-config\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-run-multus-certs\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-sys-fs\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14ca3ba3-0b3a-4063-9f59-e4f205927cfe-iptables-alerter-script\") pod \"iptables-alerter-pbjmp\" (UID: \"14ca3ba3-0b3a-4063-9f59-e4f205927cfe\") " pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb587855-58e3-4504-83e8-71afb6f79cba-cni-binary-copy\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2-konnectivity-ca\") pod \"konnectivity-agent-7v2tr\" (UID: \"4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2\") " pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-slash\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.174974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-modprobe-d\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-run\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.172998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-registration-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14ca3ba3-0b3a-4063-9f59-e4f205927cfe-iptables-alerter-script\") pod \"iptables-alerter-pbjmp\" (UID: \"14ca3ba3-0b3a-4063-9f59-e4f205927cfe\") " pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-run-netns\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-registration-dir\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-var-lib-kubelet\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-sysctl-d\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-var-lib-cni-bin\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-run-netns\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1328ac14-7eae-451f-af99-eb968a843e61-hosts-file\") pod \"node-resolver-zvggz\" (UID: \"1328ac14-7eae-451f-af99-eb968a843e61\") " pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-multus-socket-dir-parent\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.173487    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.173533    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs podName:514cfda0-541e-4b7d-91dd-dcb870c17692 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:28.67351669 +0000 UTC m=+3.012358450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs") pod "network-metrics-daemon-njtn4" (UID: "514cfda0-541e-4b7d-91dd-dcb870c17692") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-var-lib-openvswitch\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.175861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-os-release\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-var-lib-kubelet\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-kubernetes\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54-host\") pod \"node-ca-qtpfl\" (UID: \"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54\") " pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-tuned\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173949    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-slash\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14ca3ba3-0b3a-4063-9f59-e4f205927cfe-host-slash\") pod \"iptables-alerter-pbjmp\" (UID: \"14ca3ba3-0b3a-4063-9f59-e4f205927cfe\") " pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.173956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff698d8b-a442-4540-9084-615541d3bcff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.174024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-run\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.174047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-etc-modprobe-d\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.174068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb587855-58e3-4504-83e8-71afb6f79cba-host-run-multus-certs\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.174098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-sys-fs\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.174142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.174243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb587855-58e3-4504-83e8-71afb6f79cba-multus-daemon-config\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.174506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2-agent-certs\") pod \"konnectivity-agent-7v2tr\" (UID: \"4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2\") " pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.174581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-tmp\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.176594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.176272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff698d8b-a442-4540-9084-615541d3bcff-ovn-node-metrics-cert\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.180019 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.179997    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:28.180131 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.180024    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:28.180131 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.180037    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hwzbw for pod openshift-network-diagnostics/network-check-target-8vg5v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:28.180131 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.180099    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw podName:d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:28.680081206 +0000 UTC m=+3.018922979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hwzbw" (UniqueName: "kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw") pod "network-check-target-8vg5v" (UID: "d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:28.181194 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.181173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp2w7\" (UniqueName: \"kubernetes.io/projected/1328ac14-7eae-451f-af99-eb968a843e61-kube-api-access-lp2w7\") pod \"node-resolver-zvggz\" (UID: \"1328ac14-7eae-451f-af99-eb968a843e61\") " pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.183277 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.183248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvx4z\" (UniqueName: \"kubernetes.io/projected/ff698d8b-a442-4540-9084-615541d3bcff-kube-api-access-mvx4z\") pod \"ovnkube-node-d5rxk\" (UID: \"ff698d8b-a442-4540-9084-615541d3bcff\") " pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.183412 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.183284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9p9\" (UniqueName: \"kubernetes.io/projected/4e1f81e2-e886-4bbf-9b1c-e4d70201b449-kube-api-access-dw9p9\") pod \"multus-additional-cni-plugins-cgqmz\" (UID: \"4e1f81e2-e886-4bbf-9b1c-e4d70201b449\") " pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.183412 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.183359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66v4z\" (UniqueName: \"kubernetes.io/projected/e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0-kube-api-access-66v4z\") pod \"aws-ebs-csi-driver-node-2cmfb\" (UID: \"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.183685 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.183665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnhf\" (UniqueName: \"kubernetes.io/projected/514cfda0-541e-4b7d-91dd-dcb870c17692-kube-api-access-rqnhf\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:28.184202 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.184157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tng2b\" (UniqueName: \"kubernetes.io/projected/eb587855-58e3-4504-83e8-71afb6f79cba-kube-api-access-tng2b\") pod \"multus-q6tdh\" (UID: \"eb587855-58e3-4504-83e8-71afb6f79cba\") " pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.184833 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.184810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm5n5\" (UniqueName: \"kubernetes.io/projected/14ca3ba3-0b3a-4063-9f59-e4f205927cfe-kube-api-access-vm5n5\") pod \"iptables-alerter-pbjmp\" (UID: \"14ca3ba3-0b3a-4063-9f59-e4f205927cfe\") " pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.185198 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.185179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5h8m\" (UniqueName: \"kubernetes.io/projected/2bb3d2d5-8956-4fc5-b3e8-abb97c415f41-kube-api-access-n5h8m\") pod \"tuned-cjvwv\" (UID: \"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41\") " pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.185738 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.185717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnvh\" (UniqueName: \"kubernetes.io/projected/6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54-kube-api-access-ltnvh\") pod \"node-ca-qtpfl\" (UID: \"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54\") " pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.362372 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.362287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qtpfl"
Jul 24 21:15:28.366513 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.366492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:28.376247 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.376221    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q6tdh"
Jul 24 21:15:28.381980 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.381954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb"
Jul 24 21:15:28.389584 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.389564    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pbjmp"
Jul 24 21:15:28.396220 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.396200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgqmz"
Jul 24 21:15:28.401733 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.401712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:28.408436 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.408416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zvggz"
Jul 24 21:15:28.414042 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.414022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cjvwv"
Jul 24 21:15:28.475310 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.475265    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:28.675670 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.675592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:28.675670 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.675659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:28.675869 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.675745    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:28.675869 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.675757    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:28.675869 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.675820    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs podName:514cfda0-541e-4b7d-91dd-dcb870c17692 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:29.675799056 +0000 UTC m=+4.014640821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs") pod "network-metrics-daemon-njtn4" (UID: "514cfda0-541e-4b7d-91dd-dcb870c17692") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:28.675869 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.675841    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret podName:6df404c6-1f92-408c-b16b-8697d4f86fdc nodeName:}" failed. No retries permitted until 2026-07-24 21:15:29.675830339 +0000 UTC m=+4.014672097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret") pod "global-pull-secret-syncer-xc9pb" (UID: "6df404c6-1f92-408c-b16b-8697d4f86fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:28.776400 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:28.776358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:28.776575 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.776541    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:28.776575 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.776565    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:28.776650 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.776579    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hwzbw for pod openshift-network-diagnostics/network-check-target-8vg5v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:28.776650 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:28.776642    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw podName:d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:29.776627425 +0000 UTC m=+4.115469183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwzbw" (UniqueName: "kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw") pod "network-check-target-8vg5v" (UID: "d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:29.021986 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:29.021963    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb3d2d5_8956_4fc5_b3e8_abb97c415f41.slice/crio-68b4c17f740e788397ef4ea9dba7ebbb74f543d5f5d35eaa7fb20ce1acae35d2 WatchSource:0}: Error finding container 68b4c17f740e788397ef4ea9dba7ebbb74f543d5f5d35eaa7fb20ce1acae35d2: Status 404 returned error can't find the container with id 68b4c17f740e788397ef4ea9dba7ebbb74f543d5f5d35eaa7fb20ce1acae35d2
Jul 24 21:15:29.024804 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:29.024776    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1f81e2_e886_4bbf_9b1c_e4d70201b449.slice/crio-fb7c17f5b8f6b6a686d628e5030891faa8cc18a000c74540361034cdfafabd63 WatchSource:0}: Error finding container fb7c17f5b8f6b6a686d628e5030891faa8cc18a000c74540361034cdfafabd63: Status 404 returned error can't find the container with id fb7c17f5b8f6b6a686d628e5030891faa8cc18a000c74540361034cdfafabd63
Jul 24 21:15:29.025533 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:15:29.025492    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ca3ba3_0b3a_4063_9f59_e4f205927cfe.slice/crio-4315f42c31170f7b38804e2f8bd9b4baca1c00c2c64fb1899bba34461454642b WatchSource:0}: Error finding container 4315f42c31170f7b38804e2f8bd9b4baca1c00c2c64fb1899bba34461454642b: Status 404 returned error can't find the container with id 4315f42c31170f7b38804e2f8bd9b4baca1c00c2c64fb1899bba34461454642b
Jul 24 21:15:29.115226 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.115047    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 21:10:28 +0000 UTC" deadline="2028-02-11 12:11:12.989584276 +0000 UTC"
Jul 24 21:15:29.115226 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.115189    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13598h55m43.874399756s"
Jul 24 21:15:29.282014 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.281985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:29.282160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.281998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:29.282160 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.282120    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:29.282236 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.282158    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:29.289240 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.289208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cjvwv" event={"ID":"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41","Type":"ContainerStarted","Data":"68b4c17f740e788397ef4ea9dba7ebbb74f543d5f5d35eaa7fb20ce1acae35d2"}
Jul 24 21:15:29.290950 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.290931    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9789b24e4fc6e162649ac96a8871367" containerID="bc0f54941a682c183f2b7932f8cfa5273e248a8c29beeefd2f297484da63d3bc" exitCode=0
Jul 24 21:15:29.291033 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.290994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" event={"ID":"d9789b24e4fc6e162649ac96a8871367","Type":"ContainerDied","Data":"bc0f54941a682c183f2b7932f8cfa5273e248a8c29beeefd2f297484da63d3bc"}
Jul 24 21:15:29.295286 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.295260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal" event={"ID":"a258609098d82a70d5e5404dc8cfce27","Type":"ContainerStarted","Data":"ad8fcb094b862e255bcf606e3ad39dadacaba4b5e3b83f84e94308b2ef06bd3b"}
Jul 24 21:15:29.296499 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.296477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qtpfl" event={"ID":"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54","Type":"ContainerStarted","Data":"87faa236ccd0459fda798f98e7f55157fadabfe7f598af588a3fc3807f7c1fc5"}
Jul 24 21:15:29.297915 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.297884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pbjmp" event={"ID":"14ca3ba3-0b3a-4063-9f59-e4f205927cfe","Type":"ContainerStarted","Data":"4315f42c31170f7b38804e2f8bd9b4baca1c00c2c64fb1899bba34461454642b"}
Jul 24 21:15:29.300756 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.300738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgqmz" event={"ID":"4e1f81e2-e886-4bbf-9b1c-e4d70201b449","Type":"ContainerStarted","Data":"fb7c17f5b8f6b6a686d628e5030891faa8cc18a000c74540361034cdfafabd63"}
Jul 24 21:15:29.302133 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.302114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb" event={"ID":"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0","Type":"ContainerStarted","Data":"844d2496e893f884700a1b1ba1284c53770b715c57da442e41e80b4c38e8aea0"}
Jul 24 21:15:29.304094 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.304074    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerStarted","Data":"0d1ae4c15991e0505dc15c752da73b05f4b310157537a83cc889776fabe7b762"}
Jul 24 21:15:29.305328 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.305298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zvggz" event={"ID":"1328ac14-7eae-451f-af99-eb968a843e61","Type":"ContainerStarted","Data":"70aedaa72d3e7dceb14f34bbf5fab6b2d9b314ff714a720d72ea8ebf6bec7cad"}
Jul 24 21:15:29.306432 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.306412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7v2tr" event={"ID":"4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2","Type":"ContainerStarted","Data":"71e741fdced65e697f589dea62cb7a163fb55ac54d16d24ea27fa24661749baa"}
Jul 24 21:15:29.307466 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.307448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q6tdh" event={"ID":"eb587855-58e3-4504-83e8-71afb6f79cba","Type":"ContainerStarted","Data":"c484e9a00f0ae438dff8aadce9734879f2f8ed5b27ad329d7cf1e28cc17ea7e0"}
Jul 24 21:15:29.320409 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.318602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-163.ec2.internal" podStartSLOduration=2.318585877 podStartE2EDuration="2.318585877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:29.318102908 +0000 UTC m=+3.656944681" watchObservedRunningTime="2026-07-24 21:15:29.318585877 +0000 UTC m=+3.657427652"
Jul 24 21:15:29.683521 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.683422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:29.683521 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.683517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:29.683745 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.683647    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:29.683745 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.683710    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret podName:6df404c6-1f92-408c-b16b-8697d4f86fdc nodeName:}" failed. No retries permitted until 2026-07-24 21:15:31.683691641 +0000 UTC m=+6.022533400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret") pod "global-pull-secret-syncer-xc9pb" (UID: "6df404c6-1f92-408c-b16b-8697d4f86fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:29.684103 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.684084    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:29.684192 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.684136    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs podName:514cfda0-541e-4b7d-91dd-dcb870c17692 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:31.684120702 +0000 UTC m=+6.022962471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs") pod "network-metrics-daemon-njtn4" (UID: "514cfda0-541e-4b7d-91dd-dcb870c17692") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:29.787755 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:29.787718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:29.787979 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.787963    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:29.788049 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.787987    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:29.788049 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.788000    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hwzbw for pod openshift-network-diagnostics/network-check-target-8vg5v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:29.788148 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:29.788084    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw podName:d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:31.788040353 +0000 UTC m=+6.126882127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwzbw" (UniqueName: "kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw") pod "network-check-target-8vg5v" (UID: "d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:30.115944 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:30.115824    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 21:10:28 +0000 UTC" deadline="2028-01-02 01:34:06.232197557 +0000 UTC"
Jul 24 21:15:30.115944 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:30.115892    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12628h18m36.116309381s"
Jul 24 21:15:30.281944 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:30.281911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:30.282100 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:30.282059    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:30.326843 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:30.326804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" event={"ID":"d9789b24e4fc6e162649ac96a8871367","Type":"ContainerStarted","Data":"74230881e3ac011ff7df3fde018f199f6a900fd9f2d863b18e1cbb4e00fe20a9"}
Jul 24 21:15:31.281845 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:31.281807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:31.282326 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.281957    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:31.282326 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:31.282202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:31.282326 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.282291    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:31.703164 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:31.702883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:31.703164 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:31.702967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:31.703164 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.703055    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:31.703164 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.703117    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:31.703164 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.703140    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs podName:514cfda0-541e-4b7d-91dd-dcb870c17692 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:35.703118019 +0000 UTC m=+10.041959787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs") pod "network-metrics-daemon-njtn4" (UID: "514cfda0-541e-4b7d-91dd-dcb870c17692") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:31.703689 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.703173    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret podName:6df404c6-1f92-408c-b16b-8697d4f86fdc nodeName:}" failed. No retries permitted until 2026-07-24 21:15:35.703157262 +0000 UTC m=+10.041999016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret") pod "global-pull-secret-syncer-xc9pb" (UID: "6df404c6-1f92-408c-b16b-8697d4f86fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:31.803471 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:31.803432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:31.803655 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.803624    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:31.803655 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.803648    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:31.803771 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.803660    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hwzbw for pod openshift-network-diagnostics/network-check-target-8vg5v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:31.803771 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:31.803727    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw podName:d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:35.803707594 +0000 UTC m=+10.142549372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwzbw" (UniqueName: "kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw") pod "network-check-target-8vg5v" (UID: "d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:32.283752 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:32.283719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:32.284208 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:32.283846    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:33.282492 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:33.281816    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:33.282492 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:33.282023    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:33.282492 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:33.282367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:33.282492 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:33.282457    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:34.284246 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:34.283620    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:34.284246 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:34.283743    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:35.283325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:35.281648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:35.283325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:35.281685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:35.283325 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.281800    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:35.283325 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.281903    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:35.737654 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:35.737061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:35.737654 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:35.737143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:35.737654 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.737182    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:35.737654 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.737232    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:35.737654 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.737245    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret podName:6df404c6-1f92-408c-b16b-8697d4f86fdc nodeName:}" failed. No retries permitted until 2026-07-24 21:15:43.737227416 +0000 UTC m=+18.076069192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret") pod "global-pull-secret-syncer-xc9pb" (UID: "6df404c6-1f92-408c-b16b-8697d4f86fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:35.737654 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.737266    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs podName:514cfda0-541e-4b7d-91dd-dcb870c17692 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:43.7372538 +0000 UTC m=+18.076095558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs") pod "network-metrics-daemon-njtn4" (UID: "514cfda0-541e-4b7d-91dd-dcb870c17692") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:35.837933 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:35.837858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:35.838129 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.838019    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:35.838129 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.838046    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:35.838129 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.838062    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hwzbw for pod openshift-network-diagnostics/network-check-target-8vg5v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:35.838129 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:35.838124    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw podName:d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:43.838103297 +0000 UTC m=+18.176945071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwzbw" (UniqueName: "kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw") pod "network-check-target-8vg5v" (UID: "d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:36.282993 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:36.282954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:36.283133 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:36.283072    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:37.282282 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:37.281694    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:37.282282 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:37.281821    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:37.282282 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:37.282159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:37.282282 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:37.282240    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:38.281764 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:38.281732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:38.282010 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:38.281867    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:39.282282 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:39.282242    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:39.282282 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:39.282277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:39.282805 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:39.282380    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:39.282805 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:39.282541    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:40.281812 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:40.281769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:40.282011 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:40.281916    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:41.281325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:41.281284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:41.281810 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:41.281460    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:41.281885 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:41.281284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:41.281949 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:41.281920    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:42.282062 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:42.282029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:42.282547 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:42.282150    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:43.281450 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:43.281414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:43.281450 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:43.281432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:43.281718 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.281553    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:43.281718 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.281679    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:43.801402 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:43.801300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:43.801402 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:43.801378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:43.801902 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.801454    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:43.801902 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.801484    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:43.801902 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.801527    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs podName:514cfda0-541e-4b7d-91dd-dcb870c17692 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:59.801506546 +0000 UTC m=+34.140348300 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs") pod "network-metrics-daemon-njtn4" (UID: "514cfda0-541e-4b7d-91dd-dcb870c17692") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:43.801902 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.801541    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret podName:6df404c6-1f92-408c-b16b-8697d4f86fdc nodeName:}" failed. No retries permitted until 2026-07-24 21:15:59.801534792 +0000 UTC m=+34.140376546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret") pod "global-pull-secret-syncer-xc9pb" (UID: "6df404c6-1f92-408c-b16b-8697d4f86fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:43.902168 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:43.902137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:43.902367 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.902278    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:43.902367 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.902296    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:43.902367 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.902308    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hwzbw for pod openshift-network-diagnostics/network-check-target-8vg5v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:43.902367 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:43.902367    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw podName:d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:59.902354768 +0000 UTC m=+34.241196522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwzbw" (UniqueName: "kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw") pod "network-check-target-8vg5v" (UID: "d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:44.281534 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:44.281462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:44.281698 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:44.281585    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:45.282154 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:45.282124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:45.282588 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:45.282246    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:45.282588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:45.282288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:45.282588 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:45.282405    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:46.282043 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:46.282010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:46.282224 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:46.282114    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:47.281451 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:47.281434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:47.281513 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:47.281452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:47.281552 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:47.281536    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:47.281600 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:47.281587    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:48.282722 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.282355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:48.283327 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:48.282825    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:48.369201 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.369157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cjvwv" event={"ID":"2bb3d2d5-8956-4fc5-b3e8-abb97c415f41","Type":"ContainerStarted","Data":"6be3e53377f85fea7f330910ab27fa58815dfc40aa797baa2228841419f0658a"}
Jul 24 21:15:48.370757 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.370719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qtpfl" event={"ID":"6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54","Type":"ContainerStarted","Data":"ba33013786e04f5a906445341973e8d105d310437935840aec2d59c3842ea8ab"}
Jul 24 21:15:48.372456 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.372432    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e1f81e2-e886-4bbf-9b1c-e4d70201b449" containerID="a6bea71fd352615b4d09dc6597d9ee7261709291c07f0123fdb3f38de9439d92" exitCode=0
Jul 24 21:15:48.372572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.372464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgqmz" event={"ID":"4e1f81e2-e886-4bbf-9b1c-e4d70201b449","Type":"ContainerDied","Data":"a6bea71fd352615b4d09dc6597d9ee7261709291c07f0123fdb3f38de9439d92"}
Jul 24 21:15:48.374084 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.374066    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb" event={"ID":"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0","Type":"ContainerStarted","Data":"444e7e03c5da22a6699c73959792bba768d3e9316908fa8830d75daac2901532"}
Jul 24 21:15:48.377352 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.377333    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:15:48.377790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.377771    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff698d8b-a442-4540-9084-615541d3bcff" containerID="96869b000c5ec4864e38c7bf6c31407be9b8f471aaade2f208fd726f7dd46ea1" exitCode=1
Jul 24 21:15:48.377938 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.377841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerStarted","Data":"1f45b72a6cc52d9f0b906255e29b7ae4248f316b9b746f74b31a6607ccb1ba1d"}
Jul 24 21:15:48.377999 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.377917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerStarted","Data":"b956bc98ddae41284c2878ab37d3188cff3e372a9337f4e62bdafe1f7183c00a"}
Jul 24 21:15:48.378044 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.377995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerStarted","Data":"79380ddad710af92418bc9c3051b749fdd792b721ab61bb0ccba73fc7f95e798"}
Jul 24 21:15:48.378044 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.378041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerStarted","Data":"2c372fdba4a5b0b2914e13d4c81132a1ff2435c3d4dc704534d3f41abea2122d"}
Jul 24 21:15:48.378126 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.378086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerDied","Data":"96869b000c5ec4864e38c7bf6c31407be9b8f471aaade2f208fd726f7dd46ea1"}
Jul 24 21:15:48.378178 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.378137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerStarted","Data":"712e24c09e8310f3916b20310ba8f924420ebedc6d604ffa596ab11eb6d6a950"}
Jul 24 21:15:48.379330 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.379306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zvggz" event={"ID":"1328ac14-7eae-451f-af99-eb968a843e61","Type":"ContainerStarted","Data":"da5b43074c9e45c95298d807fccf4a2ddb10fc8036671c1d6a75308e8103ff5a"}
Jul 24 21:15:48.380992 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.380962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7v2tr" event={"ID":"4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2","Type":"ContainerStarted","Data":"a4b3e805e5c12542d343840ff29db42bc54e04b758bf693ae23abf7ff34b70fe"}
Jul 24 21:15:48.382526 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.382495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q6tdh" event={"ID":"eb587855-58e3-4504-83e8-71afb6f79cba","Type":"ContainerStarted","Data":"668a0773ee6df2ac56ed9334ff9b8a7d734ec9966a7c5954b07b42fca4464b71"}
Jul 24 21:15:48.388375 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.388332    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cjvwv" podStartSLOduration=4.104513801 podStartE2EDuration="22.388317188s" totalImagesPullingTime="18.283803387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:48.388033679 +0000 UTC m=+22.726875456" watchObservedRunningTime="2026-07-24 21:15:48.388317188 +0000 UTC m=+22.727158965"
Jul 24 21:15:48.388572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.388532    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-163.ec2.internal" podStartSLOduration=21.388526102 podStartE2EDuration="21.388526102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:30.342345426 +0000 UTC m=+4.681187243" watchObservedRunningTime="2026-07-24 21:15:48.388526102 +0000 UTC m=+22.727367879"
Jul 24 21:15:48.425644 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.425591    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7v2tr" podStartSLOduration=4.224150755 podStartE2EDuration="22.425575092s" totalImagesPullingTime="18.201424337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:48.425556276 +0000 UTC m=+22.764398051" watchObservedRunningTime="2026-07-24 21:15:48.425575092 +0000 UTC m=+22.764416934"
Jul 24 21:15:48.441617 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.441580    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zvggz" podStartSLOduration=4.167589669 podStartE2EDuration="22.441564298s" totalImagesPullingTime="18.273974629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:48.441090599 +0000 UTC m=+22.779932376" watchObservedRunningTime="2026-07-24 21:15:48.441564298 +0000 UTC m=+22.780406074"
Jul 24 21:15:48.458321 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.458278    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q6tdh" podStartSLOduration=3.5436657179999997 podStartE2EDuration="22.458262253s" totalImagesPullingTime="18.914596535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:48.457894629 +0000 UTC m=+22.796736405" watchObservedRunningTime="2026-07-24 21:15:48.458262253 +0000 UTC m=+22.797104029"
Jul 24 21:15:48.471595 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.471553    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qtpfl" podStartSLOduration=4.23107001 podStartE2EDuration="22.47153897s" totalImagesPullingTime="18.24046896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:48.471114977 +0000 UTC m=+22.809956752" watchObservedRunningTime="2026-07-24 21:15:48.47153897 +0000 UTC m=+22.810380746"
Jul 24 21:15:48.931960 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:48.931931    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 21:15:49.130643 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:49.130489    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T21:15:48.931954255Z","UUID":"d1670945-2073-4b24-8633-984fea54654b","Handler":null,"Name":"","Endpoint":""}
Jul 24 21:15:49.132323 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:49.132302    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 21:15:49.132479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:49.132348    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 21:15:49.282353 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:49.282318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:49.282353 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:49.282357    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:49.282593 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:49.282492    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:49.282679 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:49.282654    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:49.387562 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:49.387482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pbjmp" event={"ID":"14ca3ba3-0b3a-4063-9f59-e4f205927cfe","Type":"ContainerStarted","Data":"3584fcac44a2f33b466aa41af2237e47569a41a609da2d661dfa1dd1809e3500"}
Jul 24 21:15:49.389988 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:49.389958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb" event={"ID":"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0","Type":"ContainerStarted","Data":"60be17396ab6c3245c97ad74eda832867bfc8e5e08e882cc65f30a1ab661373a"}
Jul 24 21:15:49.403102 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:49.403053    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pbjmp" podStartSLOduration=5.196315689 podStartE2EDuration="23.403040036s" totalImagesPullingTime="18.206724347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:49.402847175 +0000 UTC m=+23.741688952" watchObservedRunningTime="2026-07-24 21:15:49.403040036 +0000 UTC m=+23.741881811"
Jul 24 21:15:50.281627 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:50.281596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:50.281897 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:50.281873    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:50.394203 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:50.394164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb" event={"ID":"e20b4d16-ada0-4d2e-9781-d1e59a9ee0b0","Type":"ContainerStarted","Data":"00f372309b11899fb2900d3821b7d9d331dee9dad3f291800ce0a5eb06fd17dd"}
Jul 24 21:15:50.397593 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:50.397574    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:15:50.398404 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:50.398355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerStarted","Data":"c1caad32c3253b620fa17f074d5d85c46640dc09876f8ab7041ae08be3e5f1df"}
Jul 24 21:15:50.413820 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:50.413773    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmfb" podStartSLOduration=4.054965906 podStartE2EDuration="24.413758094s" totalImagesPullingTime="20.358792188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:50.413482595 +0000 UTC m=+24.752324374" watchObservedRunningTime="2026-07-24 21:15:50.413758094 +0000 UTC m=+24.752599869"
Jul 24 21:15:51.281259 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:51.281230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:51.281259 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:51.281255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:51.281622 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:51.281354    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:51.281622 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:51.281477    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:52.281971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:52.281779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:52.282452 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:52.282079    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:52.998851 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:52.998767    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:52.999345 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:52.999323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:53.281442 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:53.281411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:53.281630 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:53.281414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:53.281630 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:53.281536    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:53.281630 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:53.281595    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:53.408091 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:53.408066    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:15:53.408862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:53.408480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerStarted","Data":"4df23a44b8a1c9f81cdb658a27aa45b2be1413c9c77ca84db3e69d653d919a49"}
Jul 24 21:15:53.408862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:53.408748    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:53.408968 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:53.408944    2580 scope.go:117] "RemoveContainer" containerID="96869b000c5ec4864e38c7bf6c31407be9b8f471aaade2f208fd726f7dd46ea1"
Jul 24 21:15:53.412492 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:53.410502    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e1f81e2-e886-4bbf-9b1c-e4d70201b449" containerID="0ee58355aadc268232f7e63f08f92dbe6577a17afe9c4561c67367baafc11b50" exitCode=0
Jul 24 21:15:53.412492 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:53.410580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgqmz" event={"ID":"4e1f81e2-e886-4bbf-9b1c-e4d70201b449","Type":"ContainerDied","Data":"0ee58355aadc268232f7e63f08f92dbe6577a17afe9c4561c67367baafc11b50"}
Jul 24 21:15:53.426828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:53.426804    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:54.281963 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.281803    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:54.282088 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:54.282040    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:54.415495 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.415411    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e1f81e2-e886-4bbf-9b1c-e4d70201b449" containerID="e6ac1beed4d2abd35083423647151b5368b2ed943f6cf2bb80834b6fa4e52ca2" exitCode=0
Jul 24 21:15:54.415495 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.415454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgqmz" event={"ID":"4e1f81e2-e886-4bbf-9b1c-e4d70201b449","Type":"ContainerDied","Data":"e6ac1beed4d2abd35083423647151b5368b2ed943f6cf2bb80834b6fa4e52ca2"}
Jul 24 21:15:54.420021 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.420000    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:15:54.420420 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.420371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" event={"ID":"ff698d8b-a442-4540-9084-615541d3bcff","Type":"ContainerStarted","Data":"aa1764e1423e38db316de0e2251dd30d0faba131cefd1a0a0d493803d63b5564"}
Jul 24 21:15:54.420502 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.420496    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:54.420740 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.420721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:54.427551 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.427517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8vg5v"]
Jul 24 21:15:54.427679 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.427670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:54.428095 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:54.427765    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:54.428419 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.428342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xc9pb"]
Jul 24 21:15:54.428699 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.428568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:54.428699 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:54.428666    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:54.429067 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.429043    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-njtn4"]
Jul 24 21:15:54.429178 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.429148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:54.429276 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:54.429253    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:54.438672 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.438645    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:54.467675 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:54.467625    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" podStartSLOduration=10.157006626 podStartE2EDuration="28.467606351s" totalImagesPullingTime="18.310599725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:54.466348411 +0000 UTC m=+28.805190187" watchObservedRunningTime="2026-07-24 21:15:54.467606351 +0000 UTC m=+28.806448120"
Jul 24 21:15:55.422925 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:55.422901    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:55.606366 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:55.606327    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:55.606540 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:55.606516    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:55.606914 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:55.606895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7v2tr"
Jul 24 21:15:56.282532 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:56.282321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:56.282697 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:56.282415    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:56.282697 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:56.282596    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:56.282697 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:56.282443    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:56.282697 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:56.282688    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:56.282848 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:56.282774    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:56.427846 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:56.427817    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e1f81e2-e886-4bbf-9b1c-e4d70201b449" containerID="51ef27ccbb3fa7688b4d13ec56b691492bc7808da7155b7a7d11db9b87801eaa" exitCode=0
Jul 24 21:15:56.428301 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:56.427905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgqmz" event={"ID":"4e1f81e2-e886-4bbf-9b1c-e4d70201b449","Type":"ContainerDied","Data":"51ef27ccbb3fa7688b4d13ec56b691492bc7808da7155b7a7d11db9b87801eaa"}
Jul 24 21:15:56.428301 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:56.428041    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:57.281329 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:57.281289    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:15:57.294635 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:57.294588    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" podUID="ff698d8b-a442-4540-9084-615541d3bcff" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 24 21:15:57.305502 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:57.305467    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" podUID="ff698d8b-a442-4540-9084-615541d3bcff" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 24 21:15:57.442180 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:57.442095    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk" podUID="ff698d8b-a442-4540-9084-615541d3bcff" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 24 21:15:58.285667 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:58.285636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:58.285867 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:58.285683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:58.285867 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:58.285749    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:58.285867 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:58.285768    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:15:58.285867 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:58.285850    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:15:58.286122 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:58.285929    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:15:59.040024 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:59.039997    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zvggz_1328ac14-7eae-451f-af99-eb968a843e61/dns-node-resolver/0.log"
Jul 24 21:15:59.800985 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:59.800958    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qtpfl_6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54/node-ca/0.log"
Jul 24 21:15:59.823310 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:59.823266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:15:59.823494 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:59.823328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:15:59.823494 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:59.823447    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:59.823580 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:59.823520    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret podName:6df404c6-1f92-408c-b16b-8697d4f86fdc nodeName:}" failed. No retries permitted until 2026-07-24 21:16:31.823504608 +0000 UTC m=+66.162346362 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret") pod "global-pull-secret-syncer-xc9pb" (UID: "6df404c6-1f92-408c-b16b-8697d4f86fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:59.823580 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:59.823451    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:59.823681 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:59.823608    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs podName:514cfda0-541e-4b7d-91dd-dcb870c17692 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:31.823587389 +0000 UTC m=+66.162429158 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs") pod "network-metrics-daemon-njtn4" (UID: "514cfda0-541e-4b7d-91dd-dcb870c17692") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:59.923672 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:15:59.923634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:15:59.923839 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:59.923820    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:59.923895 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:59.923845    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:59.923895 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:59.923857    2580 projected.go:196] Error preparing data for projected volume kube-api-access-hwzbw for pod openshift-network-diagnostics/network-check-target-8vg5v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:59.924001 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:15:59.923924    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw podName:d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:31.92390445 +0000 UTC m=+66.262746217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwzbw" (UniqueName: "kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw") pod "network-check-target-8vg5v" (UID: "d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:16:00.281407 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:00.281360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:16:00.281971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:00.281420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:16:00.281971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:00.281367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:16:00.281971 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:00.281523    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-njtn4" podUID="514cfda0-541e-4b7d-91dd-dcb870c17692"
Jul 24 21:16:00.281971 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:00.281944    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xc9pb" podUID="6df404c6-1f92-408c-b16b-8697d4f86fdc"
Jul 24 21:16:00.282185 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:00.282022    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8vg5v" podUID="d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0"
Jul 24 21:16:00.461452 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:00.461424    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-163.ec2.internal" event="NodeReady"
Jul 24 21:16:00.461598 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:00.461553    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 21:16:01.441643 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:01.441606    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e1f81e2-e886-4bbf-9b1c-e4d70201b449" containerID="a31de9af351edbd80e98d63d41f57b11bbf1c77e1ca62867c25df9b226e5f0f9" exitCode=0
Jul 24 21:16:01.442111 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:01.441670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgqmz" event={"ID":"4e1f81e2-e886-4bbf-9b1c-e4d70201b449","Type":"ContainerDied","Data":"a31de9af351edbd80e98d63d41f57b11bbf1c77e1ca62867c25df9b226e5f0f9"}
Jul 24 21:16:02.284323 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.284296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:16:02.284490 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.284473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:16:02.284490 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.284484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:16:02.288137 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.288096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 21:16:02.288137 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.288128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 21:16:02.288378 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.288348    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66lqr\""
Jul 24 21:16:02.288601 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.288584    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 21:16:02.288662 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.288621    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 21:16:02.288716 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.288675    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7jm8m\""
Jul 24 21:16:02.447163 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.447129    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e1f81e2-e886-4bbf-9b1c-e4d70201b449" containerID="e4deb61478c13ac306c50c9b17bb6b36b56cd439f1b9b3acdce56f1664df4c00" exitCode=0
Jul 24 21:16:02.447615 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:02.447166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgqmz" event={"ID":"4e1f81e2-e886-4bbf-9b1c-e4d70201b449","Type":"ContainerDied","Data":"e4deb61478c13ac306c50c9b17bb6b36b56cd439f1b9b3acdce56f1664df4c00"}
Jul 24 21:16:03.452811 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:03.452647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgqmz" event={"ID":"4e1f81e2-e886-4bbf-9b1c-e4d70201b449","Type":"ContainerStarted","Data":"5663e210d721ce848d21f001e6e72f5c539129fac680787089d998d40437f9b6"}
Jul 24 21:16:03.506491 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:03.504408    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cgqmz" podStartSLOduration=9.085330919 podStartE2EDuration="37.504369071s" totalImagesPullingTime="28.419038152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:03.503148623 +0000 UTC m=+37.841990398" watchObservedRunningTime="2026-07-24 21:16:03.504369071 +0000 UTC m=+37.843210848"
Jul 24 21:16:22.807029 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.806985    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5jb5d"]
Jul 24 21:16:22.808844 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.808823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:22.809490 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.809467    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zhzpr"]
Jul 24 21:16:22.811046 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.811029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zhzpr"
Jul 24 21:16:22.811782 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.811625    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 21:16:22.811782 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.811646    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 21:16:22.811929 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.811807    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t26tv\""
Jul 24 21:16:22.813764 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.813746    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 21:16:22.813861 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.813780    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jpr64\""
Jul 24 21:16:22.814987 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.814042    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 21:16:22.814987 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.814310    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 21:16:22.824766 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.824743    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zhzpr"]
Jul 24 21:16:22.831301 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.831276    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5jb5d"]
Jul 24 21:16:22.838907 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.838882    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-stc25"]
Jul 24 21:16:22.840939 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.840925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:22.843287 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.843268    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 21:16:22.843701 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.843680    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 21:16:22.843936 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.843921    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cfdvq\""
Jul 24 21:16:22.844051 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.844034    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 21:16:22.844184 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.844170    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 21:16:22.859182 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.859158    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-stc25"]
Jul 24 21:16:22.983184 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-crio-socket\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:22.983184 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d659a21-4d00-4e07-8a91-8e1401c69d1c-metrics-tls\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:22.983413 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mfv\" (UniqueName: \"kubernetes.io/projected/a88f4310-55f1-4b9e-9cd8-f879159e4818-kube-api-access-x2mfv\") pod \"ingress-canary-zhzpr\" (UID: \"a88f4310-55f1-4b9e-9cd8-f879159e4818\") " pod="openshift-ingress-canary/ingress-canary-zhzpr"
Jul 24 21:16:22.983413 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:22.983413 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zhj\" (UniqueName: \"kubernetes.io/projected/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-kube-api-access-h8zhj\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:22.983413 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d659a21-4d00-4e07-8a91-8e1401c69d1c-tmp-dir\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:22.983537 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-data-volume\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:22.983537 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:22.983537 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88f4310-55f1-4b9e-9cd8-f879159e4818-cert\") pod \"ingress-canary-zhzpr\" (UID: \"a88f4310-55f1-4b9e-9cd8-f879159e4818\") " pod="openshift-ingress-canary/ingress-canary-zhzpr"
Jul 24 21:16:22.983537 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t4qx\" (UniqueName: \"kubernetes.io/projected/3d659a21-4d00-4e07-8a91-8e1401c69d1c-kube-api-access-4t4qx\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:22.983537 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:22.983509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d659a21-4d00-4e07-8a91-8e1401c69d1c-config-volume\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.083875 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.083790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-data-volume\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.083875 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.083821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.083875 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.083843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88f4310-55f1-4b9e-9cd8-f879159e4818-cert\") pod \"ingress-canary-zhzpr\" (UID: \"a88f4310-55f1-4b9e-9cd8-f879159e4818\") " pod="openshift-ingress-canary/ingress-canary-zhzpr"
Jul 24 21:16:23.083875 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.083864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t4qx\" (UniqueName: \"kubernetes.io/projected/3d659a21-4d00-4e07-8a91-8e1401c69d1c-kube-api-access-4t4qx\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.084145 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.083971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d659a21-4d00-4e07-8a91-8e1401c69d1c-config-volume\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.084145 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-crio-socket\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.084145 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d659a21-4d00-4e07-8a91-8e1401c69d1c-metrics-tls\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.084145 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mfv\" (UniqueName: \"kubernetes.io/projected/a88f4310-55f1-4b9e-9cd8-f879159e4818-kube-api-access-x2mfv\") pod \"ingress-canary-zhzpr\" (UID: \"a88f4310-55f1-4b9e-9cd8-f879159e4818\") " pod="openshift-ingress-canary/ingress-canary-zhzpr"
Jul 24 21:16:23.084145 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.084382 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zhj\" (UniqueName: \"kubernetes.io/projected/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-kube-api-access-h8zhj\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.084382 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d659a21-4d00-4e07-8a91-8e1401c69d1c-tmp-dir\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.084382 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-data-volume\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.084578 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d659a21-4d00-4e07-8a91-8e1401c69d1c-tmp-dir\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.084578 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.084684 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-crio-socket\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.084740 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.084720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d659a21-4d00-4e07-8a91-8e1401c69d1c-config-volume\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.087931 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.087900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.088074 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.088055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d659a21-4d00-4e07-8a91-8e1401c69d1c-metrics-tls\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.088141 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.088088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a88f4310-55f1-4b9e-9cd8-f879159e4818-cert\") pod \"ingress-canary-zhzpr\" (UID: \"a88f4310-55f1-4b9e-9cd8-f879159e4818\") " pod="openshift-ingress-canary/ingress-canary-zhzpr"
Jul 24 21:16:23.096054 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.096032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t4qx\" (UniqueName: \"kubernetes.io/projected/3d659a21-4d00-4e07-8a91-8e1401c69d1c-kube-api-access-4t4qx\") pod \"dns-default-5jb5d\" (UID: \"3d659a21-4d00-4e07-8a91-8e1401c69d1c\") " pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.096174 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.096089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mfv\" (UniqueName: \"kubernetes.io/projected/a88f4310-55f1-4b9e-9cd8-f879159e4818-kube-api-access-x2mfv\") pod \"ingress-canary-zhzpr\" (UID: \"a88f4310-55f1-4b9e-9cd8-f879159e4818\") " pod="openshift-ingress-canary/ingress-canary-zhzpr"
Jul 24 21:16:23.096373 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.096358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zhj\" (UniqueName: \"kubernetes.io/projected/cd2c52d4-5c41-45fc-9d06-39a03c1a80cb-kube-api-access-h8zhj\") pod \"insights-runtime-extractor-stc25\" (UID: \"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb\") " pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.120355 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.120320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:23.125910 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.125891    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zhzpr"
Jul 24 21:16:23.149962 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.149933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-stc25"
Jul 24 21:16:23.295353 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.295324    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zhzpr"]
Jul 24 21:16:23.298566 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:23.298519    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88f4310_55f1_4b9e_9cd8_f879159e4818.slice/crio-8c0d5983fec73428fc99f53810794844b14dd77337230f92b2a7b5d3e19b671a WatchSource:0}: Error finding container 8c0d5983fec73428fc99f53810794844b14dd77337230f92b2a7b5d3e19b671a: Status 404 returned error can't find the container with id 8c0d5983fec73428fc99f53810794844b14dd77337230f92b2a7b5d3e19b671a
Jul 24 21:16:23.302863 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.302829    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5jb5d"]
Jul 24 21:16:23.305796 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:23.305771    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d659a21_4d00_4e07_8a91_8e1401c69d1c.slice/crio-f021873d1ad8bf9fe0bf4b291df010f6e2be6dec74e0e9718925b35217cd8c88 WatchSource:0}: Error finding container f021873d1ad8bf9fe0bf4b291df010f6e2be6dec74e0e9718925b35217cd8c88: Status 404 returned error can't find the container with id f021873d1ad8bf9fe0bf4b291df010f6e2be6dec74e0e9718925b35217cd8c88
Jul 24 21:16:23.332419 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.332372    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-stc25"]
Jul 24 21:16:23.337759 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:23.337734    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2c52d4_5c41_45fc_9d06_39a03c1a80cb.slice/crio-b4829c1f173aa2d4901d770fe635dcfe37203164da31ae31cf9f33cf1dacbaa5 WatchSource:0}: Error finding container b4829c1f173aa2d4901d770fe635dcfe37203164da31ae31cf9f33cf1dacbaa5: Status 404 returned error can't find the container with id b4829c1f173aa2d4901d770fe635dcfe37203164da31ae31cf9f33cf1dacbaa5
Jul 24 21:16:23.498235 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.498195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-stc25" event={"ID":"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb","Type":"ContainerStarted","Data":"b4829c1f173aa2d4901d770fe635dcfe37203164da31ae31cf9f33cf1dacbaa5"}
Jul 24 21:16:23.499308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.499279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zhzpr" event={"ID":"a88f4310-55f1-4b9e-9cd8-f879159e4818","Type":"ContainerStarted","Data":"8c0d5983fec73428fc99f53810794844b14dd77337230f92b2a7b5d3e19b671a"}
Jul 24 21:16:23.500364 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:23.500345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jb5d" event={"ID":"3d659a21-4d00-4e07-8a91-8e1401c69d1c","Type":"ContainerStarted","Data":"f021873d1ad8bf9fe0bf4b291df010f6e2be6dec74e0e9718925b35217cd8c88"}
Jul 24 21:16:27.443122 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:27.443088    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d5rxk"
Jul 24 21:16:29.189168 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.189089    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"]
Jul 24 21:16:29.190952 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.190935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"
Jul 24 21:16:29.193540 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.193519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jvz2m\""
Jul 24 21:16:29.193865 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.193849    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 24 21:16:29.202257 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.202232    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"]
Jul 24 21:16:29.232823 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.232790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e641a9f-5b54-441c-94b4-bda51b4fcaed-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2mq5z\" (UID: \"2e641a9f-5b54-441c-94b4-bda51b4fcaed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"
Jul 24 21:16:29.333663 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.333627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e641a9f-5b54-441c-94b4-bda51b4fcaed-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2mq5z\" (UID: \"2e641a9f-5b54-441c-94b4-bda51b4fcaed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"
Jul 24 21:16:29.333827 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:29.333750    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 24 21:16:29.333827 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:29.333803    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e641a9f-5b54-441c-94b4-bda51b4fcaed-tls-certificates podName:2e641a9f-5b54-441c-94b4-bda51b4fcaed nodeName:}" failed. No retries permitted until 2026-07-24 21:16:29.833789484 +0000 UTC m=+64.172631237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/2e641a9f-5b54-441c-94b4-bda51b4fcaed-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-2mq5z" (UID: "2e641a9f-5b54-441c-94b4-bda51b4fcaed") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 24 21:16:29.521948 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.521909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zhzpr" event={"ID":"a88f4310-55f1-4b9e-9cd8-f879159e4818","Type":"ContainerStarted","Data":"d073ff2ffaddf0ddac6773bcd3d4314e029cf974649d9bee356a777b669b96ad"}
Jul 24 21:16:29.525412 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.525362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jb5d" event={"ID":"3d659a21-4d00-4e07-8a91-8e1401c69d1c","Type":"ContainerStarted","Data":"3c1a7c5b615242953cca1e60d4a2c11b9fa57bc529353a9e3a9859d62d1afaad"}
Jul 24 21:16:29.525570 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.525440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jb5d" event={"ID":"3d659a21-4d00-4e07-8a91-8e1401c69d1c","Type":"ContainerStarted","Data":"d3e32805ab84108cfcd2acaeb63d82a33b99c81d5fa0693e00b01a888505ffc0"}
Jul 24 21:16:29.525570 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.525545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:29.527742 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.527680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-stc25" event={"ID":"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb","Type":"ContainerStarted","Data":"f84d879f0c2c13df2ebc1303da6b69489a116de538332ff30812a21fd543d682"}
Jul 24 21:16:29.541927 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.541869    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zhzpr" podStartSLOduration=1.9597740830000001 podStartE2EDuration="7.541847731s" totalImagesPullingTime="5.582073648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:29.540755621 +0000 UTC m=+63.879597408" watchObservedRunningTime="2026-07-24 21:16:29.541847731 +0000 UTC m=+63.880689508"
Jul 24 21:16:29.559848 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.559792    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5jb5d" podStartSLOduration=1.985013516 podStartE2EDuration="7.559771988s" totalImagesPullingTime="5.574758472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:29.558170262 +0000 UTC m=+63.897012040" watchObservedRunningTime="2026-07-24 21:16:29.559771988 +0000 UTC m=+63.898613767"
Jul 24 21:16:29.837450 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.837347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e641a9f-5b54-441c-94b4-bda51b4fcaed-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2mq5z\" (UID: \"2e641a9f-5b54-441c-94b4-bda51b4fcaed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"
Jul 24 21:16:29.841562 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:29.841532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2e641a9f-5b54-441c-94b4-bda51b4fcaed-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2mq5z\" (UID: \"2e641a9f-5b54-441c-94b4-bda51b4fcaed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"
Jul 24 21:16:30.099324 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:30.099247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"
Jul 24 21:16:30.242971 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:30.242939    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"]
Jul 24 21:16:30.246655 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:30.246620    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e641a9f_5b54_441c_94b4_bda51b4fcaed.slice/crio-7b2462ad2847baa64b4ed9b9b12eaf69caf59f8b7d8bb34eafc381c3d595226c WatchSource:0}: Error finding container 7b2462ad2847baa64b4ed9b9b12eaf69caf59f8b7d8bb34eafc381c3d595226c: Status 404 returned error can't find the container with id 7b2462ad2847baa64b4ed9b9b12eaf69caf59f8b7d8bb34eafc381c3d595226c
Jul 24 21:16:30.534788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:30.534749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z" event={"ID":"2e641a9f-5b54-441c-94b4-bda51b4fcaed","Type":"ContainerStarted","Data":"7b2462ad2847baa64b4ed9b9b12eaf69caf59f8b7d8bb34eafc381c3d595226c"}
Jul 24 21:16:31.853404 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.853353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:16:31.853859 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.853453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:16:31.856939 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.856906    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 21:16:31.857762 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.857725    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 21:16:31.867334 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.867307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/514cfda0-541e-4b7d-91dd-dcb870c17692-metrics-certs\") pod \"network-metrics-daemon-njtn4\" (UID: \"514cfda0-541e-4b7d-91dd-dcb870c17692\") " pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:16:31.867498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.867314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6df404c6-1f92-408c-b16b-8697d4f86fdc-original-pull-secret\") pod \"global-pull-secret-syncer-xc9pb\" (UID: \"6df404c6-1f92-408c-b16b-8697d4f86fdc\") " pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:16:31.954759 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.954719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:16:31.957748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.957717    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 21:16:31.967632 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.967608    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 21:16:31.978984 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.978956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwzbw\" (UniqueName: \"kubernetes.io/projected/d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0-kube-api-access-hwzbw\") pod \"network-check-target-8vg5v\" (UID: \"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0\") " pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:16:31.993672 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:31.993635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xc9pb"
Jul 24 21:16:32.008118 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:32.008087    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7jm8m\""
Jul 24 21:16:32.008360 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:32.008341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66lqr\""
Jul 24 21:16:32.010433 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:32.010335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:16:32.014888 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:32.014868    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-njtn4"
Jul 24 21:16:32.637776 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:32.637744    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-njtn4"]
Jul 24 21:16:32.641885 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:32.641858    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514cfda0_541e_4b7d_91dd_dcb870c17692.slice/crio-e7904c9b8cf7451b0d4db3cba047399a016ea54784b4d7e5d31a14d7d6eb4c5c WatchSource:0}: Error finding container e7904c9b8cf7451b0d4db3cba047399a016ea54784b4d7e5d31a14d7d6eb4c5c: Status 404 returned error can't find the container with id e7904c9b8cf7451b0d4db3cba047399a016ea54784b4d7e5d31a14d7d6eb4c5c
Jul 24 21:16:32.863516 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:32.863482    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xc9pb"]
Jul 24 21:16:32.867570 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:32.867544    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8vg5v"]
Jul 24 21:16:33.174833 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:33.174756    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bcfcfc_b6c6_42d3_8914_9a1db1edfde0.slice/crio-a1c5991b61fdd4e8aea75f15d6b33395e13dabce44e9d3b69edd053881c3e57f WatchSource:0}: Error finding container a1c5991b61fdd4e8aea75f15d6b33395e13dabce44e9d3b69edd053881c3e57f: Status 404 returned error can't find the container with id a1c5991b61fdd4e8aea75f15d6b33395e13dabce44e9d3b69edd053881c3e57f
Jul 24 21:16:33.175107 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:33.175091    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df404c6_1f92_408c_b16b_8697d4f86fdc.slice/crio-1eafe632c8e903c6fb70e9394028a0923bc5a7aa656f78000154b33f6f9f0126 WatchSource:0}: Error finding container 1eafe632c8e903c6fb70e9394028a0923bc5a7aa656f78000154b33f6f9f0126: Status 404 returned error can't find the container with id 1eafe632c8e903c6fb70e9394028a0923bc5a7aa656f78000154b33f6f9f0126
Jul 24 21:16:33.544828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:33.544788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xc9pb" event={"ID":"6df404c6-1f92-408c-b16b-8697d4f86fdc","Type":"ContainerStarted","Data":"1eafe632c8e903c6fb70e9394028a0923bc5a7aa656f78000154b33f6f9f0126"}
Jul 24 21:16:33.546832 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:33.546804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-stc25" event={"ID":"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb","Type":"ContainerStarted","Data":"ae947b95075cf6269ff3dc105d7a6ae670f041633ee9359505f025eec9fbf8b7"}
Jul 24 21:16:33.548445 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:33.548411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z" event={"ID":"2e641a9f-5b54-441c-94b4-bda51b4fcaed","Type":"ContainerStarted","Data":"b5f0aa9ba1f4460f66170705d4a1720ab2b1f5f00b5ec7307755e1915a7c58e2"}
Jul 24 21:16:33.548657 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:33.548603    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"
Jul 24 21:16:33.549854 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:33.549816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8vg5v" event={"ID":"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0","Type":"ContainerStarted","Data":"a1c5991b61fdd4e8aea75f15d6b33395e13dabce44e9d3b69edd053881c3e57f"}
Jul 24 21:16:33.551059 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:33.551038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-njtn4" event={"ID":"514cfda0-541e-4b7d-91dd-dcb870c17692","Type":"ContainerStarted","Data":"e7904c9b8cf7451b0d4db3cba047399a016ea54784b4d7e5d31a14d7d6eb4c5c"}
Jul 24 21:16:33.554024 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:33.553979    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z"
Jul 24 21:16:33.569896 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:33.569853    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2mq5z" podStartSLOduration=1.498433245 podStartE2EDuration="4.569813745s" totalImagesPullingTime="3.0713805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:33.568636788 +0000 UTC m=+67.907478568" watchObservedRunningTime="2026-07-24 21:16:33.569813745 +0000 UTC m=+67.908655500"
Jul 24 21:16:34.277625 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.276047    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"]
Jul 24 21:16:34.280467 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.279378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.286156 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.285957    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 21:16:34.287815 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.287412    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 21:16:34.287815 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.287639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 21:16:34.287815 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.287692    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 21:16:34.288474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.288096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-d62c2\""
Jul 24 21:16:34.288474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.288301    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 21:16:34.289122 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.289065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"]
Jul 24 21:16:34.373030 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.372996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.373203 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.373042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ljh\" (UniqueName: \"kubernetes.io/projected/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-kube-api-access-f6ljh\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.373203 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.373076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.373203 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.373132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.474070 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.474032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.474244 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.474107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.474244 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.474134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ljh\" (UniqueName: \"kubernetes.io/projected/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-kube-api-access-f6ljh\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.474244 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.474165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.475210 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.475149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.480039 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.479913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.482687 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.482639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.485886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.485770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ljh\" (UniqueName: \"kubernetes.io/projected/d1ac3543-e409-44c2-8e04-edd3a1fc2da9-kube-api-access-f6ljh\") pod \"prometheus-operator-84d69f794d-r7kxf\" (UID: \"d1ac3543-e409-44c2-8e04-edd3a1fc2da9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.557928 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.557466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-njtn4" event={"ID":"514cfda0-541e-4b7d-91dd-dcb870c17692","Type":"ContainerStarted","Data":"1e2eb1950ca6f60030164ead14979919128187548067dd9c938007219c76eb8f"}
Jul 24 21:16:34.557928 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.557531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-njtn4" event={"ID":"514cfda0-541e-4b7d-91dd-dcb870c17692","Type":"ContainerStarted","Data":"eb39cef68767fdbfefc423cc3543d2003866f0e2e52819563fd9096976ed4533"}
Jul 24 21:16:34.577449 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.577348    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-njtn4" podStartSLOduration=66.988417375 podStartE2EDuration="1m8.577324091s" totalImagesPullingTime="1.588906716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:34.576792185 +0000 UTC m=+68.915633973" watchObservedRunningTime="2026-07-24 21:16:34.577324091 +0000 UTC m=+68.916165869"
Jul 24 21:16:34.595609 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.595575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"
Jul 24 21:16:34.762456 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:34.762364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-r7kxf"]
Jul 24 21:16:34.767176 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:34.767142    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ac3543_e409_44c2_8e04_edd3a1fc2da9.slice/crio-2619b8d62a702d6e0d5269e0935622533358d13d65498c083d43e520d4a72667 WatchSource:0}: Error finding container 2619b8d62a702d6e0d5269e0935622533358d13d65498c083d43e520d4a72667: Status 404 returned error can't find the container with id 2619b8d62a702d6e0d5269e0935622533358d13d65498c083d43e520d4a72667
Jul 24 21:16:35.563438 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:35.563239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf" event={"ID":"d1ac3543-e409-44c2-8e04-edd3a1fc2da9","Type":"ContainerStarted","Data":"2619b8d62a702d6e0d5269e0935622533358d13d65498c083d43e520d4a72667"}
Jul 24 21:16:38.345422 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.345372    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-nfqff"]
Jul 24 21:16:38.348103 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.348076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-nfqff"
Jul 24 21:16:38.351135 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.351114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 21:16:38.351436 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.351409    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 21:16:38.351553 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.351537    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-m8fl2\""
Jul 24 21:16:38.368650 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.368620    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-nfqff"]
Jul 24 21:16:38.402588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.402553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vfnq\" (UniqueName: \"kubernetes.io/projected/58895569-e4cb-4d1a-8d28-576d265b1c16-kube-api-access-7vfnq\") pod \"downloads-7c97dc4555-nfqff\" (UID: \"58895569-e4cb-4d1a-8d28-576d265b1c16\") " pod="openshift-console/downloads-7c97dc4555-nfqff"
Jul 24 21:16:38.503187 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.503083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vfnq\" (UniqueName: \"kubernetes.io/projected/58895569-e4cb-4d1a-8d28-576d265b1c16-kube-api-access-7vfnq\") pod \"downloads-7c97dc4555-nfqff\" (UID: \"58895569-e4cb-4d1a-8d28-576d265b1c16\") " pod="openshift-console/downloads-7c97dc4555-nfqff"
Jul 24 21:16:38.522317 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.522288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vfnq\" (UniqueName: \"kubernetes.io/projected/58895569-e4cb-4d1a-8d28-576d265b1c16-kube-api-access-7vfnq\") pod \"downloads-7c97dc4555-nfqff\" (UID: \"58895569-e4cb-4d1a-8d28-576d265b1c16\") " pod="openshift-console/downloads-7c97dc4555-nfqff"
Jul 24 21:16:38.659034 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:38.658951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-nfqff"
Jul 24 21:16:39.538075 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:39.538047    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5jb5d"
Jul 24 21:16:40.334977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.334950    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-nfqff"]
Jul 24 21:16:40.338706 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:40.338680    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58895569_e4cb_4d1a_8d28_576d265b1c16.slice/crio-8f53eb8ed6eccb1bcc221a18ab96b2c2088250e792f143d4ba81510a95d26052 WatchSource:0}: Error finding container 8f53eb8ed6eccb1bcc221a18ab96b2c2088250e792f143d4ba81510a95d26052: Status 404 returned error can't find the container with id 8f53eb8ed6eccb1bcc221a18ab96b2c2088250e792f143d4ba81510a95d26052
Jul 24 21:16:40.581681 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.581644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xc9pb" event={"ID":"6df404c6-1f92-408c-b16b-8697d4f86fdc","Type":"ContainerStarted","Data":"d07b44dcd24ba792130dfc558f0c1e3cadae7b421c59bf16933318a1d9e61348"}
Jul 24 21:16:40.583540 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.583509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf" event={"ID":"d1ac3543-e409-44c2-8e04-edd3a1fc2da9","Type":"ContainerStarted","Data":"a43b51df679f60bab1d58485468b13dab7138007dbd83c49db49d9438a6da71a"}
Jul 24 21:16:40.583677 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.583563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf" event={"ID":"d1ac3543-e409-44c2-8e04-edd3a1fc2da9","Type":"ContainerStarted","Data":"b0fb1cb856ed616e9a57848fe570a57ffc445ec9e53f95b1905778758f69ceb4"}
Jul 24 21:16:40.585561 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.585529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-stc25" event={"ID":"cd2c52d4-5c41-45fc-9d06-39a03c1a80cb","Type":"ContainerStarted","Data":"b47f049fd9bc502ee8d58caffa4081890a7af2ad123c40142abbd111e8180e2d"}
Jul 24 21:16:40.586815 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.586784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-nfqff" event={"ID":"58895569-e4cb-4d1a-8d28-576d265b1c16","Type":"ContainerStarted","Data":"8f53eb8ed6eccb1bcc221a18ab96b2c2088250e792f143d4ba81510a95d26052"}
Jul 24 21:16:40.588149 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.588129    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8vg5v" event={"ID":"d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0","Type":"ContainerStarted","Data":"35c56f989f8412bc5912830d8a7d1b4a38d3c275d074cf0e96083e2eea587b6d"}
Jul 24 21:16:40.588237 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.588210    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:16:40.612481 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.612424    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xc9pb" podStartSLOduration=67.627836188 podStartE2EDuration="1m14.612383608s" totalImagesPullingTime="6.98454742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:40.611666394 +0000 UTC m=+74.950508169" watchObservedRunningTime="2026-07-24 21:16:40.612383608 +0000 UTC m=+74.951225362"
Jul 24 21:16:40.677950 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.677869    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8vg5v" podStartSLOduration=67.706173979 podStartE2EDuration="1m14.677849705s" totalImagesPullingTime="6.971675726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:40.677495065 +0000 UTC m=+75.016336842" watchObservedRunningTime="2026-07-24 21:16:40.677849705 +0000 UTC m=+75.016691485"
Jul 24 21:16:40.678348 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.678311    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-r7kxf" podStartSLOduration=1.284766646 podStartE2EDuration="6.678296752s" totalImagesPullingTime="5.393530106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:40.64677926 +0000 UTC m=+74.985621037" watchObservedRunningTime="2026-07-24 21:16:40.678296752 +0000 UTC m=+75.017138528"
Jul 24 21:16:40.700967 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:40.700918    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-stc25" podStartSLOduration=2.037249548 podStartE2EDuration="18.700901904s" totalImagesPullingTime="16.663652356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:16:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:40.699736113 +0000 UTC m=+75.038577898" watchObservedRunningTime="2026-07-24 21:16:40.700901904 +0000 UTC m=+75.039743680"
Jul 24 21:16:42.581289 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.581256    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"]
Jul 24 21:16:42.584514 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.584488    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.587064 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.587034    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 24 21:16:42.587302 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.587278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-687f5\""
Jul 24 21:16:42.587442 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.587310    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 24 21:16:42.595771 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.595749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"]
Jul 24 21:16:42.599182 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.599162    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"]
Jul 24 21:16:42.602328 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.602311    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.604833 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.604816    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 24 21:16:42.605089 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.605069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4v96b\""
Jul 24 21:16:42.605281 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.605258    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 24 21:16:42.605451 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.605268    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 24 21:16:42.617966 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.617932    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"]
Jul 24 21:16:42.625333 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.625302    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kxdqv"]
Jul 24 21:16:42.628601 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.628582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.633341 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.633311    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 21:16:42.633341 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.633312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 21:16:42.633679 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.633659    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6n2pd\""
Jul 24 21:16:42.633969 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.633949    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 21:16:42.634820 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.634789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.634925 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.634847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.634925 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.634888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c41544f-1c45-4e29-a279-3451ab823866-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.635042 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.634921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.635042 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.634949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5ee866b-2902-4480-a99c-9aff3bebcac4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.635042 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.634973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjhs\" (UniqueName: \"kubernetes.io/projected/b5ee866b-2902-4480-a99c-9aff3bebcac4-kube-api-access-rgjhs\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.635042 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.634999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5ee866b-2902-4480-a99c-9aff3bebcac4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.635042 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.635027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5ee866b-2902-4480-a99c-9aff3bebcac4-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.635275 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.635061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c41544f-1c45-4e29-a279-3451ab823866-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.635275 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.635085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztbg\" (UniqueName: \"kubernetes.io/projected/2c41544f-1c45-4e29-a279-3451ab823866-kube-api-access-4ztbg\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.735497 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.735689 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af882602-2d6e-4174-9886-3029be000688-node-exporter-textfile\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.735689 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af882602-2d6e-4174-9886-3029be000688-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.735689 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgj8l\" (UniqueName: \"kubernetes.io/projected/af882602-2d6e-4174-9886-3029be000688-kube-api-access-xgj8l\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.735689 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.735689 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af882602-2d6e-4174-9886-3029be000688-node-exporter-wtmp\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.735689 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af882602-2d6e-4174-9886-3029be000688-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.735689 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c41544f-1c45-4e29-a279-3451ab823866-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5ee866b-2902-4480-a99c-9aff3bebcac4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjhs\" (UniqueName: \"kubernetes.io/projected/b5ee866b-2902-4480-a99c-9aff3bebcac4-kube-api-access-rgjhs\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5ee866b-2902-4480-a99c-9aff3bebcac4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5ee866b-2902-4480-a99c-9aff3bebcac4-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af882602-2d6e-4174-9886-3029be000688-root\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af882602-2d6e-4174-9886-3029be000688-metrics-client-ca\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af882602-2d6e-4174-9886-3029be000688-node-exporter-tls\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c41544f-1c45-4e29-a279-3451ab823866-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztbg\" (UniqueName: \"kubernetes.io/projected/2c41544f-1c45-4e29-a279-3451ab823866-kube-api-access-4ztbg\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.736035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.735993    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af882602-2d6e-4174-9886-3029be000688-sys\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.736683 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:42.736660    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 24 21:16:42.736766 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:42.736735    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-tls podName:2c41544f-1c45-4e29-a279-3451ab823866 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:43.236713257 +0000 UTC m=+77.575555027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-7tc4r" (UID: "2c41544f-1c45-4e29-a279-3451ab823866") : secret "kube-state-metrics-tls" not found
Jul 24 21:16:42.737571 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.736842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c41544f-1c45-4e29-a279-3451ab823866-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.737571 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.737536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5ee866b-2902-4480-a99c-9aff3bebcac4-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.737571 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.737541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c41544f-1c45-4e29-a279-3451ab823866-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.737758 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.737738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.739546 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.739499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5ee866b-2902-4480-a99c-9aff3bebcac4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.739642 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.739580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.739642 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.739583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b5ee866b-2902-4480-a99c-9aff3bebcac4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.749583 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.749556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjhs\" (UniqueName: \"kubernetes.io/projected/b5ee866b-2902-4480-a99c-9aff3bebcac4-kube-api-access-rgjhs\") pod \"openshift-state-metrics-b8f49f5c6-mcrkx\" (UID: \"b5ee866b-2902-4480-a99c-9aff3bebcac4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:42.749687 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.749581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztbg\" (UniqueName: \"kubernetes.io/projected/2c41544f-1c45-4e29-a279-3451ab823866-kube-api-access-4ztbg\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:42.836610 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af882602-2d6e-4174-9886-3029be000688-node-exporter-wtmp\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.836610 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af882602-2d6e-4174-9886-3029be000688-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.836823 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af882602-2d6e-4174-9886-3029be000688-root\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.836823 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af882602-2d6e-4174-9886-3029be000688-metrics-client-ca\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.836823 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af882602-2d6e-4174-9886-3029be000688-node-exporter-wtmp\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.836823 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af882602-2d6e-4174-9886-3029be000688-node-exporter-tls\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.836823 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af882602-2d6e-4174-9886-3029be000688-sys\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.837056 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af882602-2d6e-4174-9886-3029be000688-node-exporter-textfile\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.837056 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af882602-2d6e-4174-9886-3029be000688-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.837056 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.836873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgj8l\" (UniqueName: \"kubernetes.io/projected/af882602-2d6e-4174-9886-3029be000688-kube-api-access-xgj8l\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.837196 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.837171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af882602-2d6e-4174-9886-3029be000688-sys\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.837280 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:42.837264    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 21:16:42.837341 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.837316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af882602-2d6e-4174-9886-3029be000688-metrics-client-ca\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.837491 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:42.837326    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af882602-2d6e-4174-9886-3029be000688-node-exporter-tls podName:af882602-2d6e-4174-9886-3029be000688 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:43.337306875 +0000 UTC m=+77.676148643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/af882602-2d6e-4174-9886-3029be000688-node-exporter-tls") pod "node-exporter-kxdqv" (UID: "af882602-2d6e-4174-9886-3029be000688") : secret "node-exporter-tls" not found
Jul 24 21:16:42.837491 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.837405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af882602-2d6e-4174-9886-3029be000688-root\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.837491 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.837439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af882602-2d6e-4174-9886-3029be000688-node-exporter-textfile\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.837819 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.837804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af882602-2d6e-4174-9886-3029be000688-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.839457 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.839428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af882602-2d6e-4174-9886-3029be000688-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.846154 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.846133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgj8l\" (UniqueName: \"kubernetes.io/projected/af882602-2d6e-4174-9886-3029be000688-kube-api-access-xgj8l\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:42.896376 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:42.896343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"
Jul 24 21:16:43.055874 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.055842    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx"]
Jul 24 21:16:43.060366 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:43.060327    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ee866b_2902_4480_a99c_9aff3bebcac4.slice/crio-d743209d4573789a195c3ce251ffdd7148e61c92fc04713eb8fd4dc3b8b6fca6 WatchSource:0}: Error finding container d743209d4573789a195c3ce251ffdd7148e61c92fc04713eb8fd4dc3b8b6fca6: Status 404 returned error can't find the container with id d743209d4573789a195c3ce251ffdd7148e61c92fc04713eb8fd4dc3b8b6fca6
Jul 24 21:16:43.241767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.241732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:43.244448 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.244419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c41544f-1c45-4e29-a279-3451ab823866-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-7tc4r\" (UID: \"2c41544f-1c45-4e29-a279-3451ab823866\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:43.342712 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.342630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af882602-2d6e-4174-9886-3029be000688-node-exporter-tls\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:43.345437 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.345410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af882602-2d6e-4174-9886-3029be000688-node-exporter-tls\") pod \"node-exporter-kxdqv\" (UID: \"af882602-2d6e-4174-9886-3029be000688\") " pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:43.513173 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.513143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"
Jul 24 21:16:43.538733 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.538698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kxdqv"
Jul 24 21:16:43.552179 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:43.552147    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf882602_2d6e_4174_9886_3029be000688.slice/crio-dca343a654ad692a37d347ce72f36ebb3070b8fe700fe2a14b2f6a3cfe76580b WatchSource:0}: Error finding container dca343a654ad692a37d347ce72f36ebb3070b8fe700fe2a14b2f6a3cfe76580b: Status 404 returned error can't find the container with id dca343a654ad692a37d347ce72f36ebb3070b8fe700fe2a14b2f6a3cfe76580b
Jul 24 21:16:43.603642 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.603536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxdqv" event={"ID":"af882602-2d6e-4174-9886-3029be000688","Type":"ContainerStarted","Data":"dca343a654ad692a37d347ce72f36ebb3070b8fe700fe2a14b2f6a3cfe76580b"}
Jul 24 21:16:43.606742 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.606701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx" event={"ID":"b5ee866b-2902-4480-a99c-9aff3bebcac4","Type":"ContainerStarted","Data":"2bd60dd0f429abafb1b35a42872edb69c5e964f61eb72bc730a3e2c06f7e2f82"}
Jul 24 21:16:43.606871 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.606771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx" event={"ID":"b5ee866b-2902-4480-a99c-9aff3bebcac4","Type":"ContainerStarted","Data":"4bb31220323a3f392719ec231a9f7559a6cfc6591492d4418040cf7f867deee3"}
Jul 24 21:16:43.606871 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.606814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx" event={"ID":"b5ee866b-2902-4480-a99c-9aff3bebcac4","Type":"ContainerStarted","Data":"d743209d4573789a195c3ce251ffdd7148e61c92fc04713eb8fd4dc3b8b6fca6"}
Jul 24 21:16:43.654098 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.654070    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66c57cb5f6-9nqbl"]
Jul 24 21:16:43.658594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.658573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.661505 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.661221    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 21:16:43.661505 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.661256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 21:16:43.662140 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.662111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 21:16:43.662377 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.662360    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 21:16:43.662472 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.662450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rj9jb\""
Jul 24 21:16:43.662646 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.662620    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 21:16:43.669823 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.669802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c57cb5f6-9nqbl"]
Jul 24 21:16:43.680135 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.680114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r"]
Jul 24 21:16:43.682159 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:43.682129    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c41544f_1c45_4e29_a279_3451ab823866.slice/crio-32113b1fc83aec002f1dca3c2bd7dd61a7ab009eb8a7735cb07312f298c7746e WatchSource:0}: Error finding container 32113b1fc83aec002f1dca3c2bd7dd61a7ab009eb8a7735cb07312f298c7746e: Status 404 returned error can't find the container with id 32113b1fc83aec002f1dca3c2bd7dd61a7ab009eb8a7735cb07312f298c7746e
Jul 24 21:16:43.746359 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.746326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-oauth-serving-cert\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.746526 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.746375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-console-config\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.746526 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.746439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj69x\" (UniqueName: \"kubernetes.io/projected/e8729086-72db-4124-86ef-57d77a760800-kube-api-access-zj69x\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.746526 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.746502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-oauth-config\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.746703 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.746556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-serving-cert\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.746703 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.746581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-service-ca\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.774347 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.774314    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:16:43.778110 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.778088    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.783055 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.783031    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 21:16:43.786632 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.786432    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tb95w\""
Jul 24 21:16:43.787422 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.786817    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 21:16:43.787422 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.786848    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 21:16:43.787422 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.787073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 21:16:43.787422 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.787170    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 21:16:43.787422 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.787263    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 21:16:43.789862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.789841    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 21:16:43.790996 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.790975    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 21:16:43.791309 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.791282    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 21:16:43.823654 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.819798    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:16:43.847130 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-oauth-serving-cert\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.847292 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-web-config\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847292 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-console-config\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.847292 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847491 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj69x\" (UniqueName: \"kubernetes.io/projected/e8729086-72db-4124-86ef-57d77a760800-kube-api-access-zj69x\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.847491 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847491 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-kube-api-access-vk257\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847702 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847702 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-oauth-config\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.847702 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847702 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-out\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-serving-cert\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.847835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-service-ca\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.847956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.847956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-oauth-serving-cert\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.848031 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.847960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-console-config\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.848777 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.848736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-service-ca\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.851010 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.850980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-serving-cert\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.851333 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.851317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-oauth-config\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.878663 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.878562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj69x\" (UniqueName: \"kubernetes.io/projected/e8729086-72db-4124-86ef-57d77a760800-kube-api-access-zj69x\") pod \"console-66c57cb5f6-9nqbl\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") " pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.948580 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.948749 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.948749 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-kube-api-access-vk257\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.948749 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.948749 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.948749 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.948749 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.948749 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.949193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-out\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.949193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.949193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.949193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-web-config\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.949193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.948901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.949938 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:43.949232    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 24 21:16:43.949938 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:43.949307    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-main-tls podName:f8cdd32b-7c79-4c2f-883d-4ed609ea061e nodeName:}" failed. No retries permitted until 2026-07-24 21:16:44.449286095 +0000 UTC m=+78.788127861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e") : secret "alertmanager-main-tls" not found
Jul 24 21:16:43.949938 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.949467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.950257 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.950230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.951515 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:16:43.951494    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-trusted-ca-bundle podName:f8cdd32b-7c79-4c2f-883d-4ed609ea061e nodeName:}" failed. No retries permitted until 2026-07-24 21:16:44.451471922 +0000 UTC m=+78.790313699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e") : configmap references non-existent config key: ca-bundle.crt
Jul 24 21:16:43.954331 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.953853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.954331 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.954293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.954787 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.954746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-out\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.956433 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.955011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.956934 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.956897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.957587 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.957555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.957771 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.957740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-web-config\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.958493 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.958470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:43.969767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.969739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:43.970182 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:43.970161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-kube-api-access-vk257\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:44.456808 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.455628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:44.456808 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.455684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:44.456808 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.456752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:44.459991 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.459942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:44.601086 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.601059    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"]
Jul 24 21:16:44.605192 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.605165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.608186 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.607983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 24 21:16:44.608186 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.607993    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 24 21:16:44.608186 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.608041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 24 21:16:44.608440 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.608353    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 24 21:16:44.608440 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.608363    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fksgr\""
Jul 24 21:16:44.610022 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.608895    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1kjhr6jnge95n\""
Jul 24 21:16:44.610022 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.608934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 24 21:16:44.614554 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.614455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r" event={"ID":"2c41544f-1c45-4e29-a279-3451ab823866","Type":"ContainerStarted","Data":"32113b1fc83aec002f1dca3c2bd7dd61a7ab009eb8a7735cb07312f298c7746e"}
Jul 24 21:16:44.620477 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.620455    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"]
Jul 24 21:16:44.660579 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.660543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.660687 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.660593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.660687 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.660626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.660687 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.660669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc70df34-6665-4357-a4ed-89ea199ff679-metrics-client-ca\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.660832 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.660698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9rv\" (UniqueName: \"kubernetes.io/projected/dc70df34-6665-4357-a4ed-89ea199ff679-kube-api-access-wd9rv\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.660832 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.660739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-grpc-tls\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.660832 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.660804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-tls\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.661035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.660832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.693108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.693082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:44.763429 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.762102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9rv\" (UniqueName: \"kubernetes.io/projected/dc70df34-6665-4357-a4ed-89ea199ff679-kube-api-access-wd9rv\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.763429 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.762157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-grpc-tls\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.763429 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.762216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-tls\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.763429 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.762242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.763429 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.762315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.763429 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.762344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.763429 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.762371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.763429 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.762433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc70df34-6665-4357-a4ed-89ea199ff679-metrics-client-ca\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.763429 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.763273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc70df34-6665-4357-a4ed-89ea199ff679-metrics-client-ca\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.767217 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.766270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.767217 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.766691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-tls\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.767217 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.767078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.767217 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.767138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.768845 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.768547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.768845 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.768559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dc70df34-6665-4357-a4ed-89ea199ff679-secret-grpc-tls\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.777548 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.775918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9rv\" (UniqueName: \"kubernetes.io/projected/dc70df34-6665-4357-a4ed-89ea199ff679-kube-api-access-wd9rv\") pod \"thanos-querier-6f6f4df5bf-d4psb\" (UID: \"dc70df34-6665-4357-a4ed-89ea199ff679\") " pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:44.778246 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.778168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c57cb5f6-9nqbl"]
Jul 24 21:16:44.908498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.908462    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:16:44.911340 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:44.911306    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cdd32b_7c79_4c2f_883d_4ed609ea061e.slice/crio-c232f62a2ed004b22484eb83fedc3f1519d9b1e562c273ec60ec1f17d08a5fe4 WatchSource:0}: Error finding container c232f62a2ed004b22484eb83fedc3f1519d9b1e562c273ec60ec1f17d08a5fe4: Status 404 returned error can't find the container with id c232f62a2ed004b22484eb83fedc3f1519d9b1e562c273ec60ec1f17d08a5fe4
Jul 24 21:16:44.922767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:44.922739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:45.083556 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.083521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"]
Jul 24 21:16:45.349592 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:45.349558    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc70df34_6665_4357_a4ed_89ea199ff679.slice/crio-25ee27036566ffab884457cba6eed990c807e356dd082792838bf9cd5eae6b95 WatchSource:0}: Error finding container 25ee27036566ffab884457cba6eed990c807e356dd082792838bf9cd5eae6b95: Status 404 returned error can't find the container with id 25ee27036566ffab884457cba6eed990c807e356dd082792838bf9cd5eae6b95
Jul 24 21:16:45.627285 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.626885    2580 generic.go:356] "Generic (PLEG): container finished" podID="af882602-2d6e-4174-9886-3029be000688" containerID="e6e328f03f1a910e915ea119bfcf7cf4ad5446105d2d88a08accc66ca731f24d" exitCode=0
Jul 24 21:16:45.627285 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.626985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxdqv" event={"ID":"af882602-2d6e-4174-9886-3029be000688","Type":"ContainerDied","Data":"e6e328f03f1a910e915ea119bfcf7cf4ad5446105d2d88a08accc66ca731f24d"}
Jul 24 21:16:45.632629 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.632596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb" event={"ID":"dc70df34-6665-4357-a4ed-89ea199ff679","Type":"ContainerStarted","Data":"25ee27036566ffab884457cba6eed990c807e356dd082792838bf9cd5eae6b95"}
Jul 24 21:16:45.636420 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.636355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c57cb5f6-9nqbl" event={"ID":"e8729086-72db-4124-86ef-57d77a760800","Type":"ContainerStarted","Data":"2494d23b06b1c244186bf4386cdf0415602cb4ba98bb00f4a2d008966d87b4bd"}
Jul 24 21:16:45.642025 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.641930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerStarted","Data":"c232f62a2ed004b22484eb83fedc3f1519d9b1e562c273ec60ec1f17d08a5fe4"}
Jul 24 21:16:45.651911 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.651070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx" event={"ID":"b5ee866b-2902-4480-a99c-9aff3bebcac4","Type":"ContainerStarted","Data":"78f6d52799d53f6dfb6cf2c8a9399a61d0bc47f7c9b226bd60c0f945ea83f4f7"}
Jul 24 21:16:45.659778 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.659562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r" event={"ID":"2c41544f-1c45-4e29-a279-3451ab823866","Type":"ContainerStarted","Data":"335b0f5db85c2b6a3aa302fcae1f5c349b17d99d8805e0520e4a9d029f707b00"}
Jul 24 21:16:45.659778 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.659727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r" event={"ID":"2c41544f-1c45-4e29-a279-3451ab823866","Type":"ContainerStarted","Data":"2c0f4c82cabf60c59c3fcef1f5aa4f86900a424433a3bfeff789ef9d75f10800"}
Jul 24 21:16:45.672235 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:45.671722    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mcrkx" podStartSLOduration=2.256858205 podStartE2EDuration="3.671704053s" totalImagesPullingTime="1.414845848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:45.669632548 +0000 UTC m=+80.008474335" watchObservedRunningTime="2026-07-24 21:16:45.671704053 +0000 UTC m=+80.010545832"
Jul 24 21:16:46.668436 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.668379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r" event={"ID":"2c41544f-1c45-4e29-a279-3451ab823866","Type":"ContainerStarted","Data":"2f1de54f95a1b00dd467cf3fe5e8d5563a71f951d94c3d344ab4cf14e229c47e"}
Jul 24 21:16:46.674499 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.674469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxdqv" event={"ID":"af882602-2d6e-4174-9886-3029be000688","Type":"ContainerStarted","Data":"cfc386bdd5a425e68f805fa3499fe8e5a546046bf495463b954e72cb85890abf"}
Jul 24 21:16:46.674661 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.674530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxdqv" event={"ID":"af882602-2d6e-4174-9886-3029be000688","Type":"ContainerStarted","Data":"6aa349c05b43e20ef5049559604da751a960d3481f31b76a203bbefd20cc6738"}
Jul 24 21:16:46.677342 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.677311    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerID="7d086c23a841b762d5d2d5489d92bbd68cf63b4e742220fc7f13282801a28a8b" exitCode=0
Jul 24 21:16:46.679105 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.679012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerDied","Data":"7d086c23a841b762d5d2d5489d92bbd68cf63b4e742220fc7f13282801a28a8b"}
Jul 24 21:16:46.691713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.691661    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-7tc4r" podStartSLOduration=2.97899956 podStartE2EDuration="4.69164097s" totalImagesPullingTime="1.71264141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:46.690084298 +0000 UTC m=+81.028926077" watchObservedRunningTime="2026-07-24 21:16:46.69164097 +0000 UTC m=+81.030482747"
Jul 24 21:16:46.717763 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.716803    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kxdqv" podStartSLOduration=3.544055455 podStartE2EDuration="4.716784466s" totalImagesPullingTime="1.064331099s" totalInitContainerRuntime="108.397912ms" isStatefulPod=true podCreationTimestamp="2026-07-24 21:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:46.716045992 +0000 UTC m=+81.054887768" watchObservedRunningTime="2026-07-24 21:16:46.716784466 +0000 UTC m=+81.055626236"
Jul 24 21:16:46.913074 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.913041    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f54757957-kqcgb"]
Jul 24 21:16:46.916807 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.916784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:46.919688 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.919662    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 24 21:16:46.920150 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.920131    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9bd8toe10sgpd\""
Jul 24 21:16:46.920354 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.920334    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 21:16:46.920533 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.920515    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9pcwp\""
Jul 24 21:16:46.920748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.920716    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 24 21:16:46.920901 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.920888    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 24 21:16:46.930090 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.930068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f54757957-kqcgb"]
Jul 24 21:16:46.986747 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.986715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2afef473-dc3f-422b-a6ab-b56afd71cf3c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:46.986747 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.986753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2afef473-dc3f-422b-a6ab-b56afd71cf3c-client-ca-bundle\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:46.986956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.986786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2afef473-dc3f-422b-a6ab-b56afd71cf3c-audit-log\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:46.986956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.986825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2afef473-dc3f-422b-a6ab-b56afd71cf3c-secret-metrics-server-client-certs\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:46.986956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.986855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2afef473-dc3f-422b-a6ab-b56afd71cf3c-metrics-server-audit-profiles\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:46.986956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.986871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqj28\" (UniqueName: \"kubernetes.io/projected/2afef473-dc3f-422b-a6ab-b56afd71cf3c-kube-api-access-nqj28\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:46.986956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:46.986911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2afef473-dc3f-422b-a6ab-b56afd71cf3c-secret-metrics-server-tls\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.088107 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.088066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2afef473-dc3f-422b-a6ab-b56afd71cf3c-secret-metrics-server-tls\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.088273 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.088131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2afef473-dc3f-422b-a6ab-b56afd71cf3c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.088273 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.088151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2afef473-dc3f-422b-a6ab-b56afd71cf3c-client-ca-bundle\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.088273 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.088182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2afef473-dc3f-422b-a6ab-b56afd71cf3c-audit-log\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.088273 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.088209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2afef473-dc3f-422b-a6ab-b56afd71cf3c-secret-metrics-server-client-certs\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.088273 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.088243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2afef473-dc3f-422b-a6ab-b56afd71cf3c-metrics-server-audit-profiles\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.088273 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.088270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqj28\" (UniqueName: \"kubernetes.io/projected/2afef473-dc3f-422b-a6ab-b56afd71cf3c-kube-api-access-nqj28\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.089273 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.089243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2afef473-dc3f-422b-a6ab-b56afd71cf3c-audit-log\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.089622 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.089599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2afef473-dc3f-422b-a6ab-b56afd71cf3c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.090522 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.090474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2afef473-dc3f-422b-a6ab-b56afd71cf3c-metrics-server-audit-profiles\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.093098 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.092972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2afef473-dc3f-422b-a6ab-b56afd71cf3c-client-ca-bundle\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.094197 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.094158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2afef473-dc3f-422b-a6ab-b56afd71cf3c-secret-metrics-server-client-certs\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.097333 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.097289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqj28\" (UniqueName: \"kubernetes.io/projected/2afef473-dc3f-422b-a6ab-b56afd71cf3c-kube-api-access-nqj28\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.112048 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.112001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2afef473-dc3f-422b-a6ab-b56afd71cf3c-secret-metrics-server-tls\") pod \"metrics-server-7f54757957-kqcgb\" (UID: \"2afef473-dc3f-422b-a6ab-b56afd71cf3c\") " pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.230458 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.230421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:16:47.371499 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.371422    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"]
Jul 24 21:16:47.375383 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.375352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:47.379312 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.378137    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 24 21:16:47.379312 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.378379    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t6m27\""
Jul 24 21:16:47.387144 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.385407    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"]
Jul 24 21:16:47.491745 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.491706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6fd70cf-883d-400a-8c9f-7c4ebe14d02f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-57d8p\" (UID: \"d6fd70cf-883d-400a-8c9f-7c4ebe14d02f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:47.491745 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.491745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trjlg\" (UniqueName: \"kubernetes.io/projected/d6fd70cf-883d-400a-8c9f-7c4ebe14d02f-kube-api-access-trjlg\") pod \"monitoring-plugin-68b7fc9574-57d8p\" (UID: \"d6fd70cf-883d-400a-8c9f-7c4ebe14d02f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:47.593798 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.593170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6fd70cf-883d-400a-8c9f-7c4ebe14d02f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-57d8p\" (UID: \"d6fd70cf-883d-400a-8c9f-7c4ebe14d02f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:47.593798 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.593216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjlg\" (UniqueName: \"kubernetes.io/projected/d6fd70cf-883d-400a-8c9f-7c4ebe14d02f-kube-api-access-trjlg\") pod \"monitoring-plugin-68b7fc9574-57d8p\" (UID: \"d6fd70cf-883d-400a-8c9f-7c4ebe14d02f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:47.606272 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.606241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6fd70cf-883d-400a-8c9f-7c4ebe14d02f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-57d8p\" (UID: \"d6fd70cf-883d-400a-8c9f-7c4ebe14d02f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:47.621095 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.621059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjlg\" (UniqueName: \"kubernetes.io/projected/d6fd70cf-883d-400a-8c9f-7c4ebe14d02f-kube-api-access-trjlg\") pod \"monitoring-plugin-68b7fc9574-57d8p\" (UID: \"d6fd70cf-883d-400a-8c9f-7c4ebe14d02f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:47.690806 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:47.690726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:48.927539 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.925502    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:16:48.944201 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.944164    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.948729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.949226    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.950286    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.950514    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.950751    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.950994    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.951180    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.951349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.951525    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.951685    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5tqf6m8rejsc3\""
Jul 24 21:16:48.952047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.951876    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 21:16:48.953339 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.953314    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 21:16:48.953685 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.953668    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 21:16:48.954483 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.954470    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 21:16:48.955055 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.955032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vq75n\""
Jul 24 21:16:48.955174 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.955032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 21:16:48.974412 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:48.974355    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 21:16:49.005770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.005736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.005929 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.005782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsfc9\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-kube-api-access-gsfc9\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.005929 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.005822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.005929 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.005849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.005956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.006748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.006609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107587 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107709 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107709 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107709 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107709 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107885 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107885 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107943 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.107943 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.108031 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.108031 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.107988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsfc9\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-kube-api-access-gsfc9\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.108239 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.108033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.108239 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.108059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.108239 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.108082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.108239 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.108109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.108239 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.108132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.108239 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.108161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.108239 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.108185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.109095 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.108921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.112126 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.110042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.112126 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.110620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.112126 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.110979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.112126 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.111138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.112126 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.111433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.112126 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.111817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.113889 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.113835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.115170 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.115132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.116042 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.116015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.116421 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.116379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.117986 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.117901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.118076 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.118022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.118469 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.118446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.118884 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.118843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.122783 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.122740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.123698 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.123675    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.125143 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.125100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.126096 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.126005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsfc9\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-kube-api-access-gsfc9\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.126441 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.126418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config\") pod \"prometheus-k8s-0\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.267913 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.267881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:49.754945 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.754691    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f54757957-kqcgb"]
Jul 24 21:16:49.779257 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.779215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:16:49.785764 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:49.785733    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"]
Jul 24 21:16:49.794517 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:16:49.794240    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fd70cf_883d_400a_8c9f_7c4ebe14d02f.slice/crio-d027b19110409d656171e70067a5504d8a5681c3654618f2572628da92d40069 WatchSource:0}: Error finding container d027b19110409d656171e70067a5504d8a5681c3654618f2572628da92d40069: Status 404 returned error can't find the container with id d027b19110409d656171e70067a5504d8a5681c3654618f2572628da92d40069
Jul 24 21:16:50.721010 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.720678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerStarted","Data":"5f9abe72b905354246065900175de4a805ef7827bc639c6a97c44fcecba3a72d"}
Jul 24 21:16:50.721010 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.720815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerStarted","Data":"af7f8028f636cba8ece0281df09b92ad11e1473ba77dd62456404474ea35459d"}
Jul 24 21:16:50.721010 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.720862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerStarted","Data":"ef51443114ed83a12cfd83a9f07d263ae4785bae914f23aafc5838e40e3d9a11"}
Jul 24 21:16:50.721010 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.720905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerStarted","Data":"607f4019ba0dd20a574537cf70bd70c98f4e214fc0128456ba1410ead1a2075b"}
Jul 24 21:16:50.721010 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.720945    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerStarted","Data":"af0667c584f4e202346e8660e643fcd090443155309a05fb0f2521ceff6988a8"}
Jul 24 21:16:50.726046 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.725222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c57cb5f6-9nqbl" event={"ID":"e8729086-72db-4124-86ef-57d77a760800","Type":"ContainerStarted","Data":"71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5"}
Jul 24 21:16:50.729712 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.729684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f54757957-kqcgb" event={"ID":"2afef473-dc3f-422b-a6ab-b56afd71cf3c","Type":"ContainerStarted","Data":"4c7aac3695c4fae1bcb43e27165d08511107744e035f4d43112de54b059f6419"}
Jul 24 21:16:50.732032 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.732006    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerID="683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4" exitCode=0
Jul 24 21:16:50.732160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.732137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerDied","Data":"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"}
Jul 24 21:16:50.732218 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.732209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerStarted","Data":"9810d3161a7883ef4dac301e565de07501df326fa9bf8b82578c97e79b69bbf0"}
Jul 24 21:16:50.736326 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.736154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb" event={"ID":"dc70df34-6665-4357-a4ed-89ea199ff679","Type":"ContainerStarted","Data":"a698ef1b0dddda49ca9782473076020960a0d230f3bc84ee11c01ba0b2f81d34"}
Jul 24 21:16:50.736326 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.736218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb" event={"ID":"dc70df34-6665-4357-a4ed-89ea199ff679","Type":"ContainerStarted","Data":"fa1d7efe5a838d4115c2b4c7dc4c61b9e613f1f5c1ee246b737ef1ca1e762cc6"}
Jul 24 21:16:50.736326 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.736261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb" event={"ID":"dc70df34-6665-4357-a4ed-89ea199ff679","Type":"ContainerStarted","Data":"b7a97f080deda497961a2f9627813a1c59d980a5c02c35ed9065918283a199ce"}
Jul 24 21:16:50.738058 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.738014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p" event={"ID":"d6fd70cf-883d-400a-8c9f-7c4ebe14d02f","Type":"ContainerStarted","Data":"d027b19110409d656171e70067a5504d8a5681c3654618f2572628da92d40069"}
Jul 24 21:16:50.753376 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:50.753315    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66c57cb5f6-9nqbl" podStartSLOduration=2.983435324 podStartE2EDuration="7.75329437s" totalImagesPullingTime="4.769859046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:50.751247034 +0000 UTC m=+85.090088811" watchObservedRunningTime="2026-07-24 21:16:50.75329437 +0000 UTC m=+85.092136150"
Jul 24 21:16:52.750780 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.750554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f54757957-kqcgb" event={"ID":"2afef473-dc3f-422b-a6ab-b56afd71cf3c","Type":"ContainerStarted","Data":"601040f2e51487f02786c465bcec912d90caf76d1583f05aa623da413b97eeaa"}
Jul 24 21:16:52.754383 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.754355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb" event={"ID":"dc70df34-6665-4357-a4ed-89ea199ff679","Type":"ContainerStarted","Data":"c98ec1d867b74d49755aecb4e09dea02c1e0f2688e16c6454ddb48896fcc1433"}
Jul 24 21:16:52.754538 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.754444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb" event={"ID":"dc70df34-6665-4357-a4ed-89ea199ff679","Type":"ContainerStarted","Data":"dc22a31a95e2d1ad32203ebe7b9168fdfeeb4fc176088adf688c110c02bc58c5"}
Jul 24 21:16:52.754538 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.754489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb" event={"ID":"dc70df34-6665-4357-a4ed-89ea199ff679","Type":"ContainerStarted","Data":"def4edc4389a5255649cc5db80071947fddc512d953a2ebb67d20f4d0379e1fa"}
Jul 24 21:16:52.754771 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.754755    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:16:52.756473 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.756446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p" event={"ID":"d6fd70cf-883d-400a-8c9f-7c4ebe14d02f","Type":"ContainerStarted","Data":"e573b6401ded1b7fc6db44066c63f10cc37f5ca546f3c4d9ff7c0df32a51c8a5"}
Jul 24 21:16:52.757191 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.756829    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:52.762960 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.762938    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p"
Jul 24 21:16:52.763237 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.763216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerStarted","Data":"727d393c3383d835bc4958e2f9ee4562e7e6529e4c46dc2dd532932e9e54b2a3"}
Jul 24 21:16:52.772249 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.772195    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f54757957-kqcgb" podStartSLOduration=4.773196415 podStartE2EDuration="6.77205691s" totalImagesPullingTime="1.998860495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:52.771448105 +0000 UTC m=+87.110289883" watchObservedRunningTime="2026-07-24 21:16:52.77205691 +0000 UTC m=+87.110898688"
Jul 24 21:16:52.808818 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.808753    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.594288063 podStartE2EDuration="9.808731024s" totalImagesPullingTime="6.198927047s" totalInitContainerRuntime="15.515914ms" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:52.806826078 +0000 UTC m=+87.145667861" watchObservedRunningTime="2026-07-24 21:16:52.808731024 +0000 UTC m=+87.147572800"
Jul 24 21:16:52.841418 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.838131    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb" podStartSLOduration=2.762303314 podStartE2EDuration="8.838106688s" totalImagesPullingTime="6.075803374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:52.832042033 +0000 UTC m=+87.170883809" watchObservedRunningTime="2026-07-24 21:16:52.838106688 +0000 UTC m=+87.176948462"
Jul 24 21:16:52.852128 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:52.851952    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-57d8p" podStartSLOduration=3.882494641 podStartE2EDuration="5.851931675s" totalImagesPullingTime="1.969437034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:52.850694136 +0000 UTC m=+87.189535913" watchObservedRunningTime="2026-07-24 21:16:52.851931675 +0000 UTC m=+87.190773452"
Jul 24 21:16:53.971141 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:53.970138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:53.971141 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:53.970771    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:53.976008 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:53.975982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:54.779903 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:54.779874    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:16:54.783783 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:16:54.783760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f6f4df5bf-d4psb"
Jul 24 21:17:02.824517 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.824473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-nfqff" event={"ID":"58895569-e4cb-4d1a-8d28-576d265b1c16","Type":"ContainerStarted","Data":"a877a299b498162a567521d96fa3f6ab993e989e6f06e8b1705fe63cf4563b58"}
Jul 24 21:17:02.825013 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.824828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-nfqff"
Jul 24 21:17:02.829150 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.829123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerStarted","Data":"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"}
Jul 24 21:17:02.829303 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.829179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerStarted","Data":"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"}
Jul 24 21:17:02.829303 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.829206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerStarted","Data":"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"}
Jul 24 21:17:02.829303 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.829229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerStarted","Data":"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"}
Jul 24 21:17:02.829303 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.829252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerStarted","Data":"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"}
Jul 24 21:17:02.829303 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.829274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerStarted","Data":"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"}
Jul 24 21:17:02.833851 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.833824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-nfqff"
Jul 24 21:17:02.846587 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.846530    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-nfqff" podStartSLOduration=3.008076536 podStartE2EDuration="24.846513173s" totalImagesPullingTime="21.838436637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:17:02.844709575 +0000 UTC m=+97.183551375" watchObservedRunningTime="2026-07-24 21:17:02.846513173 +0000 UTC m=+97.185354986"
Jul 24 21:17:02.881304 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:02.881240    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.418206322 podStartE2EDuration="14.881216999s" totalImagesPullingTime="11.429575668s" totalInitContainerRuntime="33.435009ms" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:17:02.878428995 +0000 UTC m=+97.217270791" watchObservedRunningTime="2026-07-24 21:17:02.881216999 +0000 UTC m=+97.220058776"
Jul 24 21:17:04.091522 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:04.091485    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c57cb5f6-9nqbl"]
Jul 24 21:17:04.268300 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:04.268258    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:17:07.230670 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:07.230632    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:17:07.231115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:07.230735    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:17:11.595915 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:11.595885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8vg5v"
Jul 24 21:17:27.236518 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:27.236474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:17:27.240468 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:27.240435    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f54757957-kqcgb"
Jul 24 21:17:29.119044 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.118981    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66c57cb5f6-9nqbl" podUID="e8729086-72db-4124-86ef-57d77a760800" containerName="console" containerID="cri-o://71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5" gracePeriod=15
Jul 24 21:17:29.411918 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.411887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:17:29.470149 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.470114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-oauth-serving-cert\") pod \"e8729086-72db-4124-86ef-57d77a760800\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") "
Jul 24 21:17:29.470149 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.470159    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-service-ca\") pod \"e8729086-72db-4124-86ef-57d77a760800\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") "
Jul 24 21:17:29.470418 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.470186    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj69x\" (UniqueName: \"kubernetes.io/projected/e8729086-72db-4124-86ef-57d77a760800-kube-api-access-zj69x\") pod \"e8729086-72db-4124-86ef-57d77a760800\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") "
Jul 24 21:17:29.470418 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.470205    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-oauth-config\") pod \"e8729086-72db-4124-86ef-57d77a760800\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") "
Jul 24 21:17:29.470418 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.470245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-console-config\") pod \"e8729086-72db-4124-86ef-57d77a760800\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") "
Jul 24 21:17:29.470418 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.470287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-serving-cert\") pod \"e8729086-72db-4124-86ef-57d77a760800\" (UID: \"e8729086-72db-4124-86ef-57d77a760800\") "
Jul 24 21:17:29.470724 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.470692    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-service-ca" (OuterVolumeSpecName: "service-ca") pod "e8729086-72db-4124-86ef-57d77a760800" (UID: "e8729086-72db-4124-86ef-57d77a760800"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:29.470848 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.470724    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-console-config" (OuterVolumeSpecName: "console-config") pod "e8729086-72db-4124-86ef-57d77a760800" (UID: "e8729086-72db-4124-86ef-57d77a760800"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:29.470848 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.470698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e8729086-72db-4124-86ef-57d77a760800" (UID: "e8729086-72db-4124-86ef-57d77a760800"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:17:29.472695 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.472670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e8729086-72db-4124-86ef-57d77a760800" (UID: "e8729086-72db-4124-86ef-57d77a760800"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:17:29.472793 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.472730    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e8729086-72db-4124-86ef-57d77a760800" (UID: "e8729086-72db-4124-86ef-57d77a760800"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:17:29.472853 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.472801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8729086-72db-4124-86ef-57d77a760800-kube-api-access-zj69x" (OuterVolumeSpecName: "kube-api-access-zj69x") pod "e8729086-72db-4124-86ef-57d77a760800" (UID: "e8729086-72db-4124-86ef-57d77a760800"). InnerVolumeSpecName "kube-api-access-zj69x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:17:29.571700 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.571660    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-oauth-serving-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:17:29.571700 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.571694    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-service-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:17:29.571700 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.571707    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj69x\" (UniqueName: \"kubernetes.io/projected/e8729086-72db-4124-86ef-57d77a760800-kube-api-access-zj69x\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:17:29.571934 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.571719    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-oauth-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:17:29.571934 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.571730    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8729086-72db-4124-86ef-57d77a760800-console-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:17:29.571934 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.571743    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8729086-72db-4124-86ef-57d77a760800-console-serving-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:17:29.954108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.954072    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8729086-72db-4124-86ef-57d77a760800" containerID="71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5" exitCode=0
Jul 24 21:17:29.954108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.954113    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c57cb5f6-9nqbl"
Jul 24 21:17:29.954307 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.954158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c57cb5f6-9nqbl" event={"ID":"e8729086-72db-4124-86ef-57d77a760800","Type":"ContainerDied","Data":"71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5"}
Jul 24 21:17:29.954307 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.954210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c57cb5f6-9nqbl" event={"ID":"e8729086-72db-4124-86ef-57d77a760800","Type":"ContainerDied","Data":"2494d23b06b1c244186bf4386cdf0415602cb4ba98bb00f4a2d008966d87b4bd"}
Jul 24 21:17:29.954307 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.954231    2580 scope.go:117] "RemoveContainer" containerID="71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5"
Jul 24 21:17:29.962909 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.962889    2580 scope.go:117] "RemoveContainer" containerID="71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5"
Jul 24 21:17:29.963195 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:17:29.963175    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5\": container with ID starting with 71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5 not found: ID does not exist" containerID="71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5"
Jul 24 21:17:29.963278 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.963204    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5"} err="failed to get container status \"71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5\": rpc error: code = NotFound desc = could not find container \"71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5\": container with ID starting with 71c2f36e3fe8b0aa5424c91e20eea72ea0de1a4e912a27d5e8bb461e8e12add5 not found: ID does not exist"
Jul 24 21:17:29.975559 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.975533    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c57cb5f6-9nqbl"]
Jul 24 21:17:29.979299 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:29.979277    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66c57cb5f6-9nqbl"]
Jul 24 21:17:30.289701 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:30.289669    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8729086-72db-4124-86ef-57d77a760800" path="/var/lib/kubelet/pods/e8729086-72db-4124-86ef-57d77a760800/volumes"
Jul 24 21:17:49.268607 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:49.268565    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:17:49.334538 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:49.334511    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:17:50.059410 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:17:50.059367    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:02.934082 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:02.934045    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:18:02.934604 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:02.934339    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="alertmanager" containerID="cri-o://af0667c584f4e202346e8660e643fcd090443155309a05fb0f2521ceff6988a8" gracePeriod=120
Jul 24 21:18:02.934604 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:02.934430    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy-metric" containerID="cri-o://5f9abe72b905354246065900175de4a805ef7827bc639c6a97c44fcecba3a72d" gracePeriod=120
Jul 24 21:18:02.934604 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:02.934464    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy-web" containerID="cri-o://ef51443114ed83a12cfd83a9f07d263ae4785bae914f23aafc5838e40e3d9a11" gracePeriod=120
Jul 24 21:18:02.934604 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:02.934476    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="config-reloader" containerID="cri-o://607f4019ba0dd20a574537cf70bd70c98f4e214fc0128456ba1410ead1a2075b" gracePeriod=120
Jul 24 21:18:02.934604 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:02.934528    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="prom-label-proxy" containerID="cri-o://727d393c3383d835bc4958e2f9ee4562e7e6529e4c46dc2dd532932e9e54b2a3" gracePeriod=120
Jul 24 21:18:02.934862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:02.934541    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy" containerID="cri-o://af7f8028f636cba8ece0281df09b92ad11e1473ba77dd62456404474ea35459d" gracePeriod=120
Jul 24 21:18:03.105377 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:03.105345    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerID="727d393c3383d835bc4958e2f9ee4562e7e6529e4c46dc2dd532932e9e54b2a3" exitCode=0
Jul 24 21:18:03.105377 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:03.105370    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerID="af7f8028f636cba8ece0281df09b92ad11e1473ba77dd62456404474ea35459d" exitCode=0
Jul 24 21:18:03.105377 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:03.105379    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerID="607f4019ba0dd20a574537cf70bd70c98f4e214fc0128456ba1410ead1a2075b" exitCode=0
Jul 24 21:18:03.105377 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:03.105401    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerID="af0667c584f4e202346e8660e643fcd090443155309a05fb0f2521ceff6988a8" exitCode=0
Jul 24 21:18:03.105642 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:03.105429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerDied","Data":"727d393c3383d835bc4958e2f9ee4562e7e6529e4c46dc2dd532932e9e54b2a3"}
Jul 24 21:18:03.105642 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:03.105489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerDied","Data":"af7f8028f636cba8ece0281df09b92ad11e1473ba77dd62456404474ea35459d"}
Jul 24 21:18:03.105642 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:03.105514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerDied","Data":"607f4019ba0dd20a574537cf70bd70c98f4e214fc0128456ba1410ead1a2075b"}
Jul 24 21:18:03.105642 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:03.105539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerDied","Data":"af0667c584f4e202346e8660e643fcd090443155309a05fb0f2521ceff6988a8"}
Jul 24 21:18:04.113622 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.113590    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerID="5f9abe72b905354246065900175de4a805ef7827bc639c6a97c44fcecba3a72d" exitCode=0
Jul 24 21:18:04.113622 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.113617    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerID="ef51443114ed83a12cfd83a9f07d263ae4785bae914f23aafc5838e40e3d9a11" exitCode=0
Jul 24 21:18:04.114028 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.113639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerDied","Data":"5f9abe72b905354246065900175de4a805ef7827bc639c6a97c44fcecba3a72d"}
Jul 24 21:18:04.114028 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.113692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerDied","Data":"ef51443114ed83a12cfd83a9f07d263ae4785bae914f23aafc5838e40e3d9a11"}
Jul 24 21:18:04.194201 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.194177    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:04.275736 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.275708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-main-db\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.275914 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.275746    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-trusted-ca-bundle\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.275914 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.275774    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.275914 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.275792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-web-config\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.275914 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.275809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-volume\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.275914 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.275831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-web\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.276160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276068    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:18:04.276160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.276269 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276166    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-out\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.276269 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-cluster-tls-config\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.276269 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276232    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-tls-assets\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.276433 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276308    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-main-tls\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.276433 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-metrics-client-ca\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.276433 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276368    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-kube-api-access-vk257\") pod \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\" (UID: \"f8cdd32b-7c79-4c2f-883d-4ed609ea061e\") "
Jul 24 21:18:04.276755 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:04.276821 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276771    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-main-db\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.276821 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.276794    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.277597 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.277550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:04.280216 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.280190    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:04.280322 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.280282    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-out" (OuterVolumeSpecName: "config-out") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:18:04.280418 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.280380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:04.280521 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.280480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:04.280756 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.280729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-volume" (OuterVolumeSpecName: "config-volume") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:04.280813 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.280746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-kube-api-access-vk257" (OuterVolumeSpecName: "kube-api-access-vk257") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "kube-api-access-vk257". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:18:04.281038 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.281012    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:18:04.282129 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.282101    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:04.284083 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.284059    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:04.292581 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.292556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-web-config" (OuterVolumeSpecName: "web-config") pod "f8cdd32b-7c79-4c2f-883d-4ed609ea061e" (UID: "f8cdd32b-7c79-4c2f-883d-4ed609ea061e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:04.377736 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377682    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-out\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.377736 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377729    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-cluster-tls-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.377736 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377742    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-tls-assets\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.377736 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377753    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-main-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.378018 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377765    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-metrics-client-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.378018 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377778    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-kube-api-access-vk257\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.378018 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377806    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.378018 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377821    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-web-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.378018 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377832    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-config-volume\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.378018 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377844    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:04.378018 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:04.377856    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f8cdd32b-7c79-4c2f-883d-4ed609ea061e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:05.125299 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.125263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f8cdd32b-7c79-4c2f-883d-4ed609ea061e","Type":"ContainerDied","Data":"c232f62a2ed004b22484eb83fedc3f1519d9b1e562c273ec60ec1f17d08a5fe4"}
Jul 24 21:18:05.125299 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.125284    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.125758 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.125368    2580 scope.go:117] "RemoveContainer" containerID="727d393c3383d835bc4958e2f9ee4562e7e6529e4c46dc2dd532932e9e54b2a3"
Jul 24 21:18:05.135897 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.135866    2580 scope.go:117] "RemoveContainer" containerID="5f9abe72b905354246065900175de4a805ef7827bc639c6a97c44fcecba3a72d"
Jul 24 21:18:05.143323 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.143304    2580 scope.go:117] "RemoveContainer" containerID="af7f8028f636cba8ece0281df09b92ad11e1473ba77dd62456404474ea35459d"
Jul 24 21:18:05.148675 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.148651    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:18:05.151686 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.151649    2580 scope.go:117] "RemoveContainer" containerID="ef51443114ed83a12cfd83a9f07d263ae4785bae914f23aafc5838e40e3d9a11"
Jul 24 21:18:05.153221 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.153198    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:18:05.159101 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.159081    2580 scope.go:117] "RemoveContainer" containerID="607f4019ba0dd20a574537cf70bd70c98f4e214fc0128456ba1410ead1a2075b"
Jul 24 21:18:05.166293 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.166276    2580 scope.go:117] "RemoveContainer" containerID="af0667c584f4e202346e8660e643fcd090443155309a05fb0f2521ceff6988a8"
Jul 24 21:18:05.174049 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.174027    2580 scope.go:117] "RemoveContainer" containerID="7d086c23a841b762d5d2d5489d92bbd68cf63b4e742220fc7f13282801a28a8b"
Jul 24 21:18:05.178682 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.178658    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:18:05.179134 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179113    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy-web"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179137    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy-web"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179153    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="alertmanager"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179161    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="alertmanager"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179186    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy-metric"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179194    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy-metric"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179208    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179216    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179224    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="prom-label-proxy"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179232    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="prom-label-proxy"
Jul 24 21:18:05.179241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179244    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="init-config-reloader"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179253    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="init-config-reloader"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179267    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8729086-72db-4124-86ef-57d77a760800" containerName="console"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179275    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8729086-72db-4124-86ef-57d77a760800" containerName="console"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179288    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="config-reloader"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179297    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="config-reloader"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179379    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8729086-72db-4124-86ef-57d77a760800" containerName="console"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179412    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179422    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="config-reloader"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179430    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy-web"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179440    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="kube-rbac-proxy-metric"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179449    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="prom-label-proxy"
Jul 24 21:18:05.179605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.179460    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" containerName="alertmanager"
Jul 24 21:18:05.185476 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.185459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.187918 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.187889    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 21:18:05.187918 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.187907    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 21:18:05.188057 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.187913    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tb95w\""
Jul 24 21:18:05.188057 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.187899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 21:18:05.188148 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.188134    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 21:18:05.188366 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.188349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 21:18:05.188459 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.188349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 21:18:05.188459 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.188379    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 21:18:05.188551 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.188463    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 21:18:05.192931 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.192908    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 21:18:05.195830 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.195813    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:18:05.284523 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79clv\" (UniqueName: \"kubernetes.io/projected/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-kube-api-access-79clv\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284523 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284728 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284728 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284728 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284728 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284728 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284728 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284900 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284900 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-config-out\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284900 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.284900 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-web-config\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.285014 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.284933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.385640 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-web-config\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.385827 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.385827 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79clv\" (UniqueName: \"kubernetes.io/projected/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-kube-api-access-79clv\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.385827 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.385827 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.385827 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.385827 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.385827 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.386169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.386169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.386169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.386169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-config-out\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.386169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.385994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.386442 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.386187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.387054 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.386959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.387736 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.387707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.389224 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.389081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.389224 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.389156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.389505 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.389314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.389505 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.389452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.389606 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.389569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-web-config\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.389886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.389865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-config-out\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.390123 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.390105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.390297 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.390280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.390776 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.390754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.394488 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.394468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79clv\" (UniqueName: \"kubernetes.io/projected/8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c-kube-api-access-79clv\") pod \"alertmanager-main-0\" (UID: \"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.495775 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.495739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:05.634323 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:05.634296    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:18:05.636317 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:18:05.636294    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8a96b5_e5e0_4368_ad10_e0b1f9fbe16c.slice/crio-d14be41367acc632ba2b3fe6be1de65511a1f404aa504215b1c4b13475e8087b WatchSource:0}: Error finding container d14be41367acc632ba2b3fe6be1de65511a1f404aa504215b1c4b13475e8087b: Status 404 returned error can't find the container with id d14be41367acc632ba2b3fe6be1de65511a1f404aa504215b1c4b13475e8087b
Jul 24 21:18:06.131819 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.131788    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c" containerID="837e1e1c293b660f99c148df0b52c77d06ae9b8816b96d037599f5f3b79dcd4f" exitCode=0
Jul 24 21:18:06.132195 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.131837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c","Type":"ContainerDied","Data":"837e1e1c293b660f99c148df0b52c77d06ae9b8816b96d037599f5f3b79dcd4f"}
Jul 24 21:18:06.132195 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.131887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c","Type":"ContainerStarted","Data":"d14be41367acc632ba2b3fe6be1de65511a1f404aa504215b1c4b13475e8087b"}
Jul 24 21:18:06.288371 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.288341    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8cdd32b-7c79-4c2f-883d-4ed609ea061e" path="/var/lib/kubelet/pods/f8cdd32b-7c79-4c2f-883d-4ed609ea061e/volumes"
Jul 24 21:18:06.954903 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.954862    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-576c5d977c-gtbzz"]
Jul 24 21:18:06.957467 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.957443    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:06.960416 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.960072    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 24 21:18:06.961112 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.960785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dbdnc\""
Jul 24 21:18:06.961112 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.961004    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 24 21:18:06.961112 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.961040    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 24 21:18:06.961330 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.961283    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 24 21:18:06.961655 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.961631    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 24 21:18:06.965682 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.965660    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 24 21:18:06.972782 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:06.972759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-576c5d977c-gtbzz"]
Jul 24 21:18:07.000051 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.000017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-metrics-client-ca\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.000192 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.000072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-secret-telemeter-client\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.000192 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.000105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-federate-client-tls\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.000295 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.000185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.000339 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.000300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqqs6\" (UniqueName: \"kubernetes.io/projected/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-kube-api-access-kqqs6\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.000382 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.000337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-telemeter-client-tls\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.000449 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.000381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-serving-certs-ca-bundle\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.000449 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.000428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.100991 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.100945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqqs6\" (UniqueName: \"kubernetes.io/projected/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-kube-api-access-kqqs6\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.100991 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.100994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-telemeter-client-tls\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.101247 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.101064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-serving-certs-ca-bundle\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.101247 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.101089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.101247 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.101146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-metrics-client-ca\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.101247 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.101180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-secret-telemeter-client\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.101247 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.101210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-federate-client-tls\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.101247 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.101245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.101901 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.101873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-serving-certs-ca-bundle\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.102263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.102146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-metrics-client-ca\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.102407 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.102308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.104275 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.104140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-secret-telemeter-client\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.104275 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.104194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-federate-client-tls\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.104452 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.104378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-telemeter-client-tls\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.104545 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.104524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.109074 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.109050    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqqs6\" (UniqueName: \"kubernetes.io/projected/f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0-kube-api-access-kqqs6\") pod \"telemeter-client-576c5d977c-gtbzz\" (UID: \"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0\") " pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.140382 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.140347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c","Type":"ContainerStarted","Data":"941bdbfbdbc7e5905f0f8aea6098ebea9ac9b49b43be257ef771157ba13e7f4a"}
Jul 24 21:18:07.140764 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.140449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c","Type":"ContainerStarted","Data":"310f8b7b459f263e30b2cacf1774a7f98bcba4cd0c10b4d21ce5212f1388a4fd"}
Jul 24 21:18:07.140764 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.140486    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c","Type":"ContainerStarted","Data":"26ea7057b5ae30467fe6612251b54bd2f7999a5f88b9145c9ecf8d6f8a612b01"}
Jul 24 21:18:07.140764 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.140513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c","Type":"ContainerStarted","Data":"15ff6f6ff8ea81b54aea09d00ff662a1a225007a8e5a7c2c3512b9bd7ea9bfc8"}
Jul 24 21:18:07.140764 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.140545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c","Type":"ContainerStarted","Data":"f65e0975098c754e9c58a4a5268b4a40b002b552ea4d31f040857ad282ee41ca"}
Jul 24 21:18:07.140764 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.140582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c","Type":"ContainerStarted","Data":"f3b270b7732a5c9b3f7c09f6f137e24c7e9e2516ced4c49feda4b2f9232e3ee0"}
Jul 24 21:18:07.167133 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.167076    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.147517795 podStartE2EDuration="2.167055549s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.537754ms" isStatefulPod=false podCreationTimestamp="2026-07-24 21:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:18:07.165476446 +0000 UTC m=+161.504318224" watchObservedRunningTime="2026-07-24 21:18:07.167055549 +0000 UTC m=+161.505897329"
Jul 24 21:18:07.258631 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.258547    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:07.258871 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.258844    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="prometheus" containerID="cri-o://35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489" gracePeriod=600
Jul 24 21:18:07.258955 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.258889    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="config-reloader" containerID="cri-o://8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894" gracePeriod=600
Jul 24 21:18:07.258955 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.258889    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="thanos-sidecar" containerID="cri-o://54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b" gracePeriod=600
Jul 24 21:18:07.258955 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.258930    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc" gracePeriod=600
Jul 24 21:18:07.259103 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.258868    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy" containerID="cri-o://2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3" gracePeriod=600
Jul 24 21:18:07.259103 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.258950    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy-web" containerID="cri-o://c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466" gracePeriod=600
Jul 24 21:18:07.272990 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.272642    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz"
Jul 24 21:18:07.437523 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.437500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-576c5d977c-gtbzz"]
Jul 24 21:18:07.439602 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:18:07.439556    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c96c61_0201_4d3a_ae0a_dedb7d9b22e0.slice/crio-b35f46af19804d6b489a80d1c2c14412e1cc70ee0deba5bc95976d56d027cd1f WatchSource:0}: Error finding container b35f46af19804d6b489a80d1c2c14412e1cc70ee0deba5bc95976d56d027cd1f: Status 404 returned error can't find the container with id b35f46af19804d6b489a80d1c2c14412e1cc70ee0deba5bc95976d56d027cd1f
Jul 24 21:18:07.529444 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.529419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:07.605769 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-tls\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.605769 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.605992 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-serving-certs-ca-bundle\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.605992 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-kubelet-serving-ca-bundle\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.605992 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605843    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsfc9\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-kube-api-access-gsfc9\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.605992 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-web-config\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.605992 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-db\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.605992 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605933    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-trusted-ca-bundle\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.605992 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605969    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-thanos-prometheus-http-client-file\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.605992 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.605993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-0\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.606359 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.606020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-tls-assets\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.606359 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.606050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.606359 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.606093    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-metrics-client-certs\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.606359 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.606121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-grpc-tls\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.606359 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.606218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:07.606359 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.606263    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:07.607362 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.607028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.607362 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.607080    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-1\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.607362 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.607115    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config-out\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.607362 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.607157    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-metrics-client-ca\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.607362 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.607193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-kube-rbac-proxy\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.607362 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.607227    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-2\") pod \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\" (UID: \"b0a82374-7e57-4f37-858d-fe4c5b8819b9\") "
Jul 24 21:18:07.608182 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.607918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:18:07.608876 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.608287    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.608876 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.608314    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.608876 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.608331    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-db\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.608876 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.608621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:07.609364 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.609333    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:07.609546 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.609523    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:07.609631 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.609594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config" (OuterVolumeSpecName: "config") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:07.609780 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.609741    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:07.610740 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.610704    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:07.610925 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.610881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:07.610998 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.610951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:07.611101 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.611063    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-kube-api-access-gsfc9" (OuterVolumeSpecName: "kube-api-access-gsfc9") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "kube-api-access-gsfc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:18:07.611231 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.611094    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:07.611722 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.611697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:18:07.611876 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.611837    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:07.611974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.611888    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:07.611974 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.611914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config-out" (OuterVolumeSpecName: "config-out") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:18:07.612244 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.612221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:07.613557 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.613536    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:07.622886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.622866    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-web-config" (OuterVolumeSpecName: "web-config") pod "b0a82374-7e57-4f37-858d-fe4c5b8819b9" (UID: "b0a82374-7e57-4f37-858d-fe4c5b8819b9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:07.709080 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709033    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709080 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709074    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709080 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709088    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-tls-assets\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709080 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709096    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709105    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-metrics-client-certs\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709116    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-grpc-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709125    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709133    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709142    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config-out\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709150    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-configmap-metrics-client-ca\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709158    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-kube-rbac-proxy\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709168    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709176    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709184    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709192    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsfc9\" (UniqueName: \"kubernetes.io/projected/b0a82374-7e57-4f37-858d-fe4c5b8819b9-kube-api-access-gsfc9\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709200    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b0a82374-7e57-4f37-858d-fe4c5b8819b9-web-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:07.709343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:07.709208    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0a82374-7e57-4f37-858d-fe4c5b8819b9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:18:08.155956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.155926    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerID="b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc" exitCode=0
Jul 24 21:18:08.155956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.155952    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerID="2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3" exitCode=0
Jul 24 21:18:08.155956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.155964    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerID="c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466" exitCode=0
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.155972    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerID="54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b" exitCode=0
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.155978    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerID="8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894" exitCode=0
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.155983    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerID="35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489" exitCode=0
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.156017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerDied","Data":"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"}
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.156045    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.156146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerDied","Data":"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"}
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.156193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerDied","Data":"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"}
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.156236    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerDied","Data":"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"}
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.156218    2580 scope.go:117] "RemoveContainer" containerID="b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.156275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerDied","Data":"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"}
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.156316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerDied","Data":"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"}
Jul 24 21:18:08.156474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.156356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b0a82374-7e57-4f37-858d-fe4c5b8819b9","Type":"ContainerDied","Data":"9810d3161a7883ef4dac301e565de07501df326fa9bf8b82578c97e79b69bbf0"}
Jul 24 21:18:08.157885 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.157854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz" event={"ID":"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0","Type":"ContainerStarted","Data":"b35f46af19804d6b489a80d1c2c14412e1cc70ee0deba5bc95976d56d027cd1f"}
Jul 24 21:18:08.166193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.166175    2580 scope.go:117] "RemoveContainer" containerID="2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"
Jul 24 21:18:08.174778 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.174756    2580 scope.go:117] "RemoveContainer" containerID="c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"
Jul 24 21:18:08.182945 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.182912    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:08.184444 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.184421    2580 scope.go:117] "RemoveContainer" containerID="54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"
Jul 24 21:18:08.187468 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.187442    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:08.193490 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.193470    2580 scope.go:117] "RemoveContainer" containerID="8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"
Jul 24 21:18:08.201326 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.201304    2580 scope.go:117] "RemoveContainer" containerID="35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"
Jul 24 21:18:08.209103 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.209086    2580 scope.go:117] "RemoveContainer" containerID="683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"
Jul 24 21:18:08.213763 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.213742    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:08.214056 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214040    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy-web"
Jul 24 21:18:08.214108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214058    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy-web"
Jul 24 21:18:08.214108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214072    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy-thanos"
Jul 24 21:18:08.214108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214080    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy-thanos"
Jul 24 21:18:08.214108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214090    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="thanos-sidecar"
Jul 24 21:18:08.214108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214095    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="thanos-sidecar"
Jul 24 21:18:08.214108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214106    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="prometheus"
Jul 24 21:18:08.214108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214111    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="prometheus"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214121    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="config-reloader"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214126    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="config-reloader"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214135    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="init-config-reloader"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214140    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="init-config-reloader"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214147    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214152    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214205    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy-web"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214215    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="prometheus"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214222    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="thanos-sidecar"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214228    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy-thanos"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214234    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="config-reloader"
Jul 24 21:18:08.214308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.214239    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" containerName="kube-rbac-proxy"
Jul 24 21:18:08.217073 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.217051    2580 scope.go:117] "RemoveContainer" containerID="b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"
Jul 24 21:18:08.217359 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:18:08.217340    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": container with ID starting with b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc not found: ID does not exist" containerID="b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"
Jul 24 21:18:08.217439 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.217369    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"} err="failed to get container status \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": rpc error: code = NotFound desc = could not find container \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": container with ID starting with b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc not found: ID does not exist"
Jul 24 21:18:08.217439 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.217405    2580 scope.go:117] "RemoveContainer" containerID="2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"
Jul 24 21:18:08.217601 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.217585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.218192 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:18:08.217977    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": container with ID starting with 2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3 not found: ID does not exist" containerID="2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"
Jul 24 21:18:08.218192 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.218024    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"} err="failed to get container status \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": rpc error: code = NotFound desc = could not find container \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": container with ID starting with 2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3 not found: ID does not exist"
Jul 24 21:18:08.218192 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.218049    2580 scope.go:117] "RemoveContainer" containerID="c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"
Jul 24 21:18:08.218366 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:18:08.218309    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": container with ID starting with c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466 not found: ID does not exist" containerID="c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"
Jul 24 21:18:08.218366 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.218339    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"} err="failed to get container status \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": rpc error: code = NotFound desc = could not find container \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": container with ID starting with c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466 not found: ID does not exist"
Jul 24 21:18:08.218366 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.218361    2580 scope.go:117] "RemoveContainer" containerID="54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"
Jul 24 21:18:08.218655 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:18:08.218639    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": container with ID starting with 54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b not found: ID does not exist" containerID="54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"
Jul 24 21:18:08.218707 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.218661    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"} err="failed to get container status \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": rpc error: code = NotFound desc = could not find container \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": container with ID starting with 54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b not found: ID does not exist"
Jul 24 21:18:08.218707 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.218675    2580 scope.go:117] "RemoveContainer" containerID="8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"
Jul 24 21:18:08.218898 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:18:08.218882    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": container with ID starting with 8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894 not found: ID does not exist" containerID="8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"
Jul 24 21:18:08.219000 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.218904    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"} err="failed to get container status \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": rpc error: code = NotFound desc = could not find container \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": container with ID starting with 8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894 not found: ID does not exist"
Jul 24 21:18:08.219000 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.218929    2580 scope.go:117] "RemoveContainer" containerID="35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"
Jul 24 21:18:08.219203 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:18:08.219183    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": container with ID starting with 35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489 not found: ID does not exist" containerID="35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"
Jul 24 21:18:08.219338 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.219204    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"} err="failed to get container status \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": rpc error: code = NotFound desc = could not find container \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": container with ID starting with 35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489 not found: ID does not exist"
Jul 24 21:18:08.219338 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.219218    2580 scope.go:117] "RemoveContainer" containerID="683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"
Jul 24 21:18:08.219517 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:18:08.219498    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": container with ID starting with 683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4 not found: ID does not exist" containerID="683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"
Jul 24 21:18:08.219573 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.219516    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"} err="failed to get container status \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": rpc error: code = NotFound desc = could not find container \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": container with ID starting with 683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4 not found: ID does not exist"
Jul 24 21:18:08.219573 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.219531    2580 scope.go:117] "RemoveContainer" containerID="b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"
Jul 24 21:18:08.219818 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.219796    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"} err="failed to get container status \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": rpc error: code = NotFound desc = could not find container \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": container with ID starting with b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc not found: ID does not exist"
Jul 24 21:18:08.219898 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.219822    2580 scope.go:117] "RemoveContainer" containerID="2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"
Jul 24 21:18:08.220179 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220085    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"} err="failed to get container status \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": rpc error: code = NotFound desc = could not find container \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": container with ID starting with 2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3 not found: ID does not exist"
Jul 24 21:18:08.220179 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220109    2580 scope.go:117] "RemoveContainer" containerID="c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"
Jul 24 21:18:08.220179 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220111    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 21:18:08.220179 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220156    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 21:18:08.220179 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220178    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 21:18:08.220584 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220478    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"} err="failed to get container status \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": rpc error: code = NotFound desc = could not find container \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": container with ID starting with c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466 not found: ID does not exist"
Jul 24 21:18:08.220584 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220500    2580 scope.go:117] "RemoveContainer" containerID="54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"
Jul 24 21:18:08.220680 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220632    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 21:18:08.220732 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 21:18:08.220790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220734    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 21:18:08.220790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5tqf6m8rejsc3\""
Jul 24 21:18:08.220887 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220697    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 21:18:08.220887 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.220815    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 21:18:08.221100 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221075    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"} err="failed to get container status \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": rpc error: code = NotFound desc = could not find container \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": container with ID starting with 54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b not found: ID does not exist"
Jul 24 21:18:08.221193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221102    2580 scope.go:117] "RemoveContainer" containerID="8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"
Jul 24 21:18:08.221260 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221226    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 21:18:08.221260 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 21:18:08.221354 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221301    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vq75n\""
Jul 24 21:18:08.221354 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221348    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 21:18:08.221455 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221366    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"} err="failed to get container status \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": rpc error: code = NotFound desc = could not find container \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": container with ID starting with 8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894 not found: ID does not exist"
Jul 24 21:18:08.221511 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221495    2580 scope.go:117] "RemoveContainer" containerID="35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"
Jul 24 21:18:08.221782 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221759    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 21:18:08.221857 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221780    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"} err="failed to get container status \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": rpc error: code = NotFound desc = could not find container \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": container with ID starting with 35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489 not found: ID does not exist"
Jul 24 21:18:08.221857 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.221797    2580 scope.go:117] "RemoveContainer" containerID="683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"
Jul 24 21:18:08.222165 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.222137    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"} err="failed to get container status \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": rpc error: code = NotFound desc = could not find container \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": container with ID starting with 683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4 not found: ID does not exist"
Jul 24 21:18:08.222254 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.222165    2580 scope.go:117] "RemoveContainer" containerID="b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"
Jul 24 21:18:08.222572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.222552    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"} err="failed to get container status \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": rpc error: code = NotFound desc = could not find container \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": container with ID starting with b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc not found: ID does not exist"
Jul 24 21:18:08.222572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.222571    2580 scope.go:117] "RemoveContainer" containerID="2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"
Jul 24 21:18:08.222810 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.222787    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"} err="failed to get container status \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": rpc error: code = NotFound desc = could not find container \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": container with ID starting with 2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3 not found: ID does not exist"
Jul 24 21:18:08.222886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.222814    2580 scope.go:117] "RemoveContainer" containerID="c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"
Jul 24 21:18:08.223520 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.223494    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"} err="failed to get container status \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": rpc error: code = NotFound desc = could not find container \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": container with ID starting with c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466 not found: ID does not exist"
Jul 24 21:18:08.223629 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.223520    2580 scope.go:117] "RemoveContainer" containerID="54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"
Jul 24 21:18:08.223873 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.223836    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"} err="failed to get container status \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": rpc error: code = NotFound desc = could not find container \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": container with ID starting with 54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b not found: ID does not exist"
Jul 24 21:18:08.223873 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.223864    2580 scope.go:117] "RemoveContainer" containerID="8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"
Jul 24 21:18:08.224007 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.223995    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 21:18:08.224157 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.224117    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"} err="failed to get container status \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": rpc error: code = NotFound desc = could not find container \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": container with ID starting with 8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894 not found: ID does not exist"
Jul 24 21:18:08.224243 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.224160    2580 scope.go:117] "RemoveContainer" containerID="35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"
Jul 24 21:18:08.224439 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.224403    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"} err="failed to get container status \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": rpc error: code = NotFound desc = could not find container \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": container with ID starting with 35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489 not found: ID does not exist"
Jul 24 21:18:08.224439 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.224439    2580 scope.go:117] "RemoveContainer" containerID="683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"
Jul 24 21:18:08.224756 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.224714    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"} err="failed to get container status \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": rpc error: code = NotFound desc = could not find container \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": container with ID starting with 683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4 not found: ID does not exist"
Jul 24 21:18:08.224756 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.224749    2580 scope.go:117] "RemoveContainer" containerID="b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"
Jul 24 21:18:08.225012 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.224995    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"} err="failed to get container status \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": rpc error: code = NotFound desc = could not find container \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": container with ID starting with b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc not found: ID does not exist"
Jul 24 21:18:08.225074 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.225013    2580 scope.go:117] "RemoveContainer" containerID="2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"
Jul 24 21:18:08.225463 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.225341    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"} err="failed to get container status \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": rpc error: code = NotFound desc = could not find container \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": container with ID starting with 2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3 not found: ID does not exist"
Jul 24 21:18:08.225463 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.225370    2580 scope.go:117] "RemoveContainer" containerID="c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"
Jul 24 21:18:08.225638 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.225613    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"} err="failed to get container status \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": rpc error: code = NotFound desc = could not find container \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": container with ID starting with c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466 not found: ID does not exist"
Jul 24 21:18:08.225638 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.225636    2580 scope.go:117] "RemoveContainer" containerID="54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"
Jul 24 21:18:08.225919 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.225899    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"} err="failed to get container status \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": rpc error: code = NotFound desc = could not find container \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": container with ID starting with 54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b not found: ID does not exist"
Jul 24 21:18:08.225995 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.225922    2580 scope.go:117] "RemoveContainer" containerID="8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"
Jul 24 21:18:08.226177 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.226155    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"} err="failed to get container status \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": rpc error: code = NotFound desc = could not find container \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": container with ID starting with 8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894 not found: ID does not exist"
Jul 24 21:18:08.226258 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.226181    2580 scope.go:117] "RemoveContainer" containerID="35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"
Jul 24 21:18:08.226494 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.226469    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"} err="failed to get container status \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": rpc error: code = NotFound desc = could not find container \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": container with ID starting with 35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489 not found: ID does not exist"
Jul 24 21:18:08.226566 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.226497    2580 scope.go:117] "RemoveContainer" containerID="683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"
Jul 24 21:18:08.226807 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.226781    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"} err="failed to get container status \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": rpc error: code = NotFound desc = could not find container \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": container with ID starting with 683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4 not found: ID does not exist"
Jul 24 21:18:08.226881 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.226810    2580 scope.go:117] "RemoveContainer" containerID="b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"
Jul 24 21:18:08.227131 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.227084    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"} err="failed to get container status \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": rpc error: code = NotFound desc = could not find container \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": container with ID starting with b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc not found: ID does not exist"
Jul 24 21:18:08.227195 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.227134    2580 scope.go:117] "RemoveContainer" containerID="2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"
Jul 24 21:18:08.227249 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.227194    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 21:18:08.227444 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.227417    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"} err="failed to get container status \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": rpc error: code = NotFound desc = could not find container \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": container with ID starting with 2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3 not found: ID does not exist"
Jul 24 21:18:08.227444 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.227443    2580 scope.go:117] "RemoveContainer" containerID="c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"
Jul 24 21:18:08.227677 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.227658    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"} err="failed to get container status \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": rpc error: code = NotFound desc = could not find container \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": container with ID starting with c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466 not found: ID does not exist"
Jul 24 21:18:08.227722 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.227679    2580 scope.go:117] "RemoveContainer" containerID="54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"
Jul 24 21:18:08.227904 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.227887    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"} err="failed to get container status \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": rpc error: code = NotFound desc = could not find container \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": container with ID starting with 54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b not found: ID does not exist"
Jul 24 21:18:08.227978 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.227906    2580 scope.go:117] "RemoveContainer" containerID="8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"
Jul 24 21:18:08.228102 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.228087    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"} err="failed to get container status \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": rpc error: code = NotFound desc = could not find container \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": container with ID starting with 8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894 not found: ID does not exist"
Jul 24 21:18:08.228167 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.228104    2580 scope.go:117] "RemoveContainer" containerID="35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"
Jul 24 21:18:08.228348 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.228333    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"} err="failed to get container status \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": rpc error: code = NotFound desc = could not find container \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": container with ID starting with 35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489 not found: ID does not exist"
Jul 24 21:18:08.228441 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.228350    2580 scope.go:117] "RemoveContainer" containerID="683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"
Jul 24 21:18:08.228577 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.228561    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"} err="failed to get container status \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": rpc error: code = NotFound desc = could not find container \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": container with ID starting with 683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4 not found: ID does not exist"
Jul 24 21:18:08.228640 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.228579    2580 scope.go:117] "RemoveContainer" containerID="b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"
Jul 24 21:18:08.228809 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.228791    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc"} err="failed to get container status \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": rpc error: code = NotFound desc = could not find container \"b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc\": container with ID starting with b913caed9d781c1c3d23ef03780bf5145ac8b5050e10464da7e4bc3f8755bafc not found: ID does not exist"
Jul 24 21:18:08.228851 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.228812    2580 scope.go:117] "RemoveContainer" containerID="2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"
Jul 24 21:18:08.229047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.229029    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3"} err="failed to get container status \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": rpc error: code = NotFound desc = could not find container \"2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3\": container with ID starting with 2b0722a5e61c945f9b4bf13463e7a1b17f1db210a9ed2094749cf7fcd05653a3 not found: ID does not exist"
Jul 24 21:18:08.229107 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.229048    2580 scope.go:117] "RemoveContainer" containerID="c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"
Jul 24 21:18:08.229273 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.229255    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466"} err="failed to get container status \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": rpc error: code = NotFound desc = could not find container \"c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466\": container with ID starting with c3eb2ae49f48e8258f5898b8581c6c804fb44abf67d27e71b18b0d57ff677466 not found: ID does not exist"
Jul 24 21:18:08.229325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.229274    2580 scope.go:117] "RemoveContainer" containerID="54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"
Jul 24 21:18:08.229526 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.229510    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b"} err="failed to get container status \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": rpc error: code = NotFound desc = could not find container \"54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b\": container with ID starting with 54a4d40c55414ef9c43b299dab3844c0759b170c2fe1fc083a444ab4bd475d7b not found: ID does not exist"
Jul 24 21:18:08.229568 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.229528    2580 scope.go:117] "RemoveContainer" containerID="8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"
Jul 24 21:18:08.229747 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.229728    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894"} err="failed to get container status \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": rpc error: code = NotFound desc = could not find container \"8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894\": container with ID starting with 8908650b166bb93af6493290349f0ba28c802e60132bffc2da817ecca1452894 not found: ID does not exist"
Jul 24 21:18:08.229821 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.229750    2580 scope.go:117] "RemoveContainer" containerID="35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"
Jul 24 21:18:08.230009 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.229977    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489"} err="failed to get container status \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": rpc error: code = NotFound desc = could not find container \"35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489\": container with ID starting with 35094c167bc711719d51cbbb10b57fd1e408351b90a91a343f128a1e452c7489 not found: ID does not exist"
Jul 24 21:18:08.230049 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.230014    2580 scope.go:117] "RemoveContainer" containerID="683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"
Jul 24 21:18:08.230311 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.230291    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4"} err="failed to get container status \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": rpc error: code = NotFound desc = could not find container \"683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4\": container with ID starting with 683670be6125b5c4bead7773a606235fd9bd7861454d33709646d97822a68dd4 not found: ID does not exist"
Jul 24 21:18:08.233032 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.233010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:08.288151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.288116    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a82374-7e57-4f37-858d-fe4c5b8819b9" path="/var/lib/kubelet/pods/b0a82374-7e57-4f37-858d-fe4c5b8819b9/volumes"
Jul 24 21:18:08.314576 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-config\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.314768 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73735a04-4aaa-4d93-b216-8f8f9c019174-config-out\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.314768 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.314768 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx54h\" (UniqueName: \"kubernetes.io/projected/73735a04-4aaa-4d93-b216-8f8f9c019174-kube-api-access-wx54h\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.314768 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.314768 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315004 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315004 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73735a04-4aaa-4d93-b216-8f8f9c019174-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315004 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315004 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315004 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315004 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.314959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.315007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.315081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.315125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.315149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.315173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-web-config\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.315209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.315236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.315595 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.315280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416000 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.415958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416000 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73735a04-4aaa-4d93-b216-8f8f9c019174-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416238 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416238 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416238 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416238 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416482 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416482 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416482 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416633 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416633 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-web-config\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416633 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416633 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416633 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416633 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-config\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416986 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73735a04-4aaa-4d93-b216-8f8f9c019174-config-out\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416986 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416986 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx54h\" (UniqueName: \"kubernetes.io/projected/73735a04-4aaa-4d93-b216-8f8f9c019174-kube-api-access-wx54h\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416986 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.416986 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.416801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.417272 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.417244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.417368 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.417346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.418151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.417847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.418151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.418117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.419641 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.419614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.420086 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.419891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.420086 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.419907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.420606 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.420450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.421259 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.420851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.421259 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.420929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73735a04-4aaa-4d93-b216-8f8f9c019174-config-out\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.421259 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.420984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.422019 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.421681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.422019 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.421979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.422553 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.422532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73735a04-4aaa-4d93-b216-8f8f9c019174-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.422734 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.422601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.424045 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.424022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.424158 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.424076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-config\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.424682 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.424660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73735a04-4aaa-4d93-b216-8f8f9c019174-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.425115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.425090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73735a04-4aaa-4d93-b216-8f8f9c019174-web-config\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.428063 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.428041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx54h\" (UniqueName: \"kubernetes.io/projected/73735a04-4aaa-4d93-b216-8f8f9c019174-kube-api-access-wx54h\") pod \"prometheus-k8s-0\" (UID: \"73735a04-4aaa-4d93-b216-8f8f9c019174\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.530426 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.530359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:08.690991 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:08.690964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:08.693282 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:18:08.693255    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73735a04_4aaa_4d93_b216_8f8f9c019174.slice/crio-82465f5273e44dd601e76bcfa97117456c2348dbbaceafdbec7e34bc671d183e WatchSource:0}: Error finding container 82465f5273e44dd601e76bcfa97117456c2348dbbaceafdbec7e34bc671d183e: Status 404 returned error can't find the container with id 82465f5273e44dd601e76bcfa97117456c2348dbbaceafdbec7e34bc671d183e
Jul 24 21:18:09.166000 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:09.165962    2580 generic.go:356] "Generic (PLEG): container finished" podID="73735a04-4aaa-4d93-b216-8f8f9c019174" containerID="d2f6cecdbf1ca6c747170acb31d7821b443b0ef26ec4b7f8703e236ca22221ee" exitCode=0
Jul 24 21:18:09.166442 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:09.166023    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73735a04-4aaa-4d93-b216-8f8f9c019174","Type":"ContainerDied","Data":"d2f6cecdbf1ca6c747170acb31d7821b443b0ef26ec4b7f8703e236ca22221ee"}
Jul 24 21:18:09.166442 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:09.166080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73735a04-4aaa-4d93-b216-8f8f9c019174","Type":"ContainerStarted","Data":"82465f5273e44dd601e76bcfa97117456c2348dbbaceafdbec7e34bc671d183e"}
Jul 24 21:18:10.173246 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.173211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz" event={"ID":"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0","Type":"ContainerStarted","Data":"2d5369f38a71ef640720c4712cfaeafcd9385735f15d3af5a374cfbae507fe69"}
Jul 24 21:18:10.173703 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.173273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz" event={"ID":"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0","Type":"ContainerStarted","Data":"d4739109ca67e0da42f36ff3804d0a751fab1510cd1532904d09d60a83fcd694"}
Jul 24 21:18:10.173703 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.173294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz" event={"ID":"f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0","Type":"ContainerStarted","Data":"48d9a7966d754f8e2a00f945e3562bbe3b5e1262e8d4b1bf0f442348ae911b76"}
Jul 24 21:18:10.176280 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.176259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73735a04-4aaa-4d93-b216-8f8f9c019174","Type":"ContainerStarted","Data":"5c96f61f21d2606e2f217a67955050728928f7d23246f932591b5680c8cfacdd"}
Jul 24 21:18:10.176410 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.176307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73735a04-4aaa-4d93-b216-8f8f9c019174","Type":"ContainerStarted","Data":"c9af41e1dcbb09d63be00fe554d8860e0ea4f5fe23f1ac75e7e412b9fa1914b5"}
Jul 24 21:18:10.176410 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.176334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73735a04-4aaa-4d93-b216-8f8f9c019174","Type":"ContainerStarted","Data":"0783734d851bbd049526dba59769c0bf969fc9dc6b0c93592c3cfe65aab37016"}
Jul 24 21:18:10.176410 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.176357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73735a04-4aaa-4d93-b216-8f8f9c019174","Type":"ContainerStarted","Data":"e2591adbc383397d9ac002ec8311390da32a1d6fc6c04cfda35edfef4f78c09c"}
Jul 24 21:18:10.176410 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.176378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73735a04-4aaa-4d93-b216-8f8f9c019174","Type":"ContainerStarted","Data":"09ac2df812921f3412ea3c9219d8b73eb2f28d2cd074ad9d63999e1ac2be4c0e"}
Jul 24 21:18:10.176540 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.176420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73735a04-4aaa-4d93-b216-8f8f9c019174","Type":"ContainerStarted","Data":"79cea49bd872c3c16a61cb89afe268916fa04c324452064fd3eb3040f5bd6c55"}
Jul 24 21:18:10.196818 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.196764    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-576c5d977c-gtbzz" podStartSLOduration=2.238199298 podStartE2EDuration="4.196747111s" totalImagesPullingTime="1.958547813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:18:10.194593724 +0000 UTC m=+164.533435500" watchObservedRunningTime="2026-07-24 21:18:10.196747111 +0000 UTC m=+164.535588886"
Jul 24 21:18:10.222274 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:10.222218    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.19824876 podStartE2EDuration="2.222201949s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.953189ms" isStatefulPod=false podCreationTimestamp="2026-07-24 21:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:18:10.220953045 +0000 UTC m=+164.559794822" watchObservedRunningTime="2026-07-24 21:18:10.222201949 +0000 UTC m=+164.561044121"
Jul 24 21:18:13.531314 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:18:13.531261    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:19:08.531590 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:19:08.531545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:19:08.547901 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:19:08.547869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:19:09.430674 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:19:09.430650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:20:26.073155 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:20:26.073127    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:20:26.079775 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:20:26.079741    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 21:20:26.081020 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:20:26.081004    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:22:34.364459 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.364428    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-b8v9m"]
Jul 24 21:22:34.367615 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.367600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:22:34.370048 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.370026    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 24 21:22:34.372548 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.372528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 24 21:22:34.373365 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.373332    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 24 21:22:34.373365 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.373341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d9dxm\""
Jul 24 21:22:34.379334 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.379313    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-b8v9m"]
Jul 24 21:22:34.418547 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.418515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5cae5431-2d94-44fb-8a3b-0fac7974678c-data\") pod \"seaweedfs-64568bcd49-b8v9m\" (UID: \"5cae5431-2d94-44fb-8a3b-0fac7974678c\") " pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:22:34.418699 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.418555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2ff\" (UniqueName: \"kubernetes.io/projected/5cae5431-2d94-44fb-8a3b-0fac7974678c-kube-api-access-qd2ff\") pod \"seaweedfs-64568bcd49-b8v9m\" (UID: \"5cae5431-2d94-44fb-8a3b-0fac7974678c\") " pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:22:34.519116 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.519072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5cae5431-2d94-44fb-8a3b-0fac7974678c-data\") pod \"seaweedfs-64568bcd49-b8v9m\" (UID: \"5cae5431-2d94-44fb-8a3b-0fac7974678c\") " pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:22:34.519307 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.519127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2ff\" (UniqueName: \"kubernetes.io/projected/5cae5431-2d94-44fb-8a3b-0fac7974678c-kube-api-access-qd2ff\") pod \"seaweedfs-64568bcd49-b8v9m\" (UID: \"5cae5431-2d94-44fb-8a3b-0fac7974678c\") " pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:22:34.519474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.519455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5cae5431-2d94-44fb-8a3b-0fac7974678c-data\") pod \"seaweedfs-64568bcd49-b8v9m\" (UID: \"5cae5431-2d94-44fb-8a3b-0fac7974678c\") " pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:22:34.536604 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.536575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2ff\" (UniqueName: \"kubernetes.io/projected/5cae5431-2d94-44fb-8a3b-0fac7974678c-kube-api-access-qd2ff\") pod \"seaweedfs-64568bcd49-b8v9m\" (UID: \"5cae5431-2d94-44fb-8a3b-0fac7974678c\") " pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:22:34.677308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.677201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:22:34.812310 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.812289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-b8v9m"]
Jul 24 21:22:34.814588 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:22:34.814561    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cae5431_2d94_44fb_8a3b_0fac7974678c.slice/crio-989ca0cf75652ca125ca18981aa532b5fdfb0fe3c03cd7e94ac0a5d24044729e WatchSource:0}: Error finding container 989ca0cf75652ca125ca18981aa532b5fdfb0fe3c03cd7e94ac0a5d24044729e: Status 404 returned error can't find the container with id 989ca0cf75652ca125ca18981aa532b5fdfb0fe3c03cd7e94ac0a5d24044729e
Jul 24 21:22:34.815880 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:34.815863    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:22:35.243609 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:35.243577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-b8v9m" event={"ID":"5cae5431-2d94-44fb-8a3b-0fac7974678c","Type":"ContainerStarted","Data":"989ca0cf75652ca125ca18981aa532b5fdfb0fe3c03cd7e94ac0a5d24044729e"}
Jul 24 21:22:38.264175 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:38.264084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-b8v9m" event={"ID":"5cae5431-2d94-44fb-8a3b-0fac7974678c","Type":"ContainerStarted","Data":"e5d53f332f4ed24f439eec8cbc381278847f3d5ef06e4c9f1b1bd3914ef5afc7"}
Jul 24 21:22:38.264645 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:38.264231    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:22:38.282338 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:38.282287    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-b8v9m" podStartSLOduration=1.241023612 podStartE2EDuration="4.282271654s" totalImagesPullingTime="3.041248042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:22:38.281447901 +0000 UTC m=+432.620289677" watchObservedRunningTime="2026-07-24 21:22:38.282271654 +0000 UTC m=+432.621113430"
Jul 24 21:22:44.270293 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:22:44.270260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-b8v9m"
Jul 24 21:23:51.303251 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.303217    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-smgwt"]
Jul 24 21:23:51.306771 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.306746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:23:51.310010 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.309990    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 24 21:23:51.310934 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.310918    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-86gk7\""
Jul 24 21:23:51.317953 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.317933    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-smgwt"]
Jul 24 21:23:51.384104 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.384072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c07919-5eaa-4979-a6fe-5a9c6b41d882-tls-certs\") pod \"model-serving-api-7b954ffc95-smgwt\" (UID: \"87c07919-5eaa-4979-a6fe-5a9c6b41d882\") " pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:23:51.384104 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.384105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdz7m\" (UniqueName: \"kubernetes.io/projected/87c07919-5eaa-4979-a6fe-5a9c6b41d882-kube-api-access-xdz7m\") pod \"model-serving-api-7b954ffc95-smgwt\" (UID: \"87c07919-5eaa-4979-a6fe-5a9c6b41d882\") " pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:23:51.484551 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.484520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c07919-5eaa-4979-a6fe-5a9c6b41d882-tls-certs\") pod \"model-serving-api-7b954ffc95-smgwt\" (UID: \"87c07919-5eaa-4979-a6fe-5a9c6b41d882\") " pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:23:51.484551 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.484553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdz7m\" (UniqueName: \"kubernetes.io/projected/87c07919-5eaa-4979-a6fe-5a9c6b41d882-kube-api-access-xdz7m\") pod \"model-serving-api-7b954ffc95-smgwt\" (UID: \"87c07919-5eaa-4979-a6fe-5a9c6b41d882\") " pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:23:51.487098 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.487074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c07919-5eaa-4979-a6fe-5a9c6b41d882-tls-certs\") pod \"model-serving-api-7b954ffc95-smgwt\" (UID: \"87c07919-5eaa-4979-a6fe-5a9c6b41d882\") " pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:23:51.494328 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.494302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdz7m\" (UniqueName: \"kubernetes.io/projected/87c07919-5eaa-4979-a6fe-5a9c6b41d882-kube-api-access-xdz7m\") pod \"model-serving-api-7b954ffc95-smgwt\" (UID: \"87c07919-5eaa-4979-a6fe-5a9c6b41d882\") " pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:23:51.618594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.618516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:23:51.757626 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:51.757594    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-smgwt"]
Jul 24 21:23:51.760741 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:23:51.760706    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c07919_5eaa_4979_a6fe_5a9c6b41d882.slice/crio-a315aa8903c87ab8b84fec1af3460e771aed0e00fb441ca0a303f45c7a68ef44 WatchSource:0}: Error finding container a315aa8903c87ab8b84fec1af3460e771aed0e00fb441ca0a303f45c7a68ef44: Status 404 returned error can't find the container with id a315aa8903c87ab8b84fec1af3460e771aed0e00fb441ca0a303f45c7a68ef44
Jul 24 21:23:52.567951 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:52.567916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-smgwt" event={"ID":"87c07919-5eaa-4979-a6fe-5a9c6b41d882","Type":"ContainerStarted","Data":"a315aa8903c87ab8b84fec1af3460e771aed0e00fb441ca0a303f45c7a68ef44"}
Jul 24 21:23:54.578318 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:54.578281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-smgwt" event={"ID":"87c07919-5eaa-4979-a6fe-5a9c6b41d882","Type":"ContainerStarted","Data":"dbacac250580bf93b965db862707ba96716f7f7917f3021b9bb07126447c3df8"}
Jul 24 21:23:54.578738 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:54.578372    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:23:54.596484 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:23:54.596435    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-smgwt" podStartSLOduration=1.465959524 podStartE2EDuration="3.596417642s" totalImagesPullingTime="2.130458118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:23:54.594935003 +0000 UTC m=+508.933776779" watchObservedRunningTime="2026-07-24 21:23:54.596417642 +0000 UTC m=+508.935259446"
Jul 24 21:24:05.587237 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:05.587203    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-smgwt"
Jul 24 21:24:07.256900 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.256866    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-4dq9z"]
Jul 24 21:24:07.263749 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.263727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4dq9z"
Jul 24 21:24:07.275586 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.275563    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4dq9z"]
Jul 24 21:24:07.329883 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.329847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63df2c9-0e80-449d-9fa8-9945622619f6-model-cache\") pod \"s3-init-4dq9z\" (UID: \"d63df2c9-0e80-449d-9fa8-9945622619f6\") " pod="kserve/s3-init-4dq9z"
Jul 24 21:24:07.329883 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.329890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkcl\" (UniqueName: \"kubernetes.io/projected/d63df2c9-0e80-449d-9fa8-9945622619f6-kube-api-access-7kkcl\") pod \"s3-init-4dq9z\" (UID: \"d63df2c9-0e80-449d-9fa8-9945622619f6\") " pod="kserve/s3-init-4dq9z"
Jul 24 21:24:07.430590 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.430561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63df2c9-0e80-449d-9fa8-9945622619f6-model-cache\") pod \"s3-init-4dq9z\" (UID: \"d63df2c9-0e80-449d-9fa8-9945622619f6\") " pod="kserve/s3-init-4dq9z"
Jul 24 21:24:07.430759 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.430600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkcl\" (UniqueName: \"kubernetes.io/projected/d63df2c9-0e80-449d-9fa8-9945622619f6-kube-api-access-7kkcl\") pod \"s3-init-4dq9z\" (UID: \"d63df2c9-0e80-449d-9fa8-9945622619f6\") " pod="kserve/s3-init-4dq9z"
Jul 24 21:24:07.430934 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.430917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63df2c9-0e80-449d-9fa8-9945622619f6-model-cache\") pod \"s3-init-4dq9z\" (UID: \"d63df2c9-0e80-449d-9fa8-9945622619f6\") " pod="kserve/s3-init-4dq9z"
Jul 24 21:24:07.442039 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.442013    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkcl\" (UniqueName: \"kubernetes.io/projected/d63df2c9-0e80-449d-9fa8-9945622619f6-kube-api-access-7kkcl\") pod \"s3-init-4dq9z\" (UID: \"d63df2c9-0e80-449d-9fa8-9945622619f6\") " pod="kserve/s3-init-4dq9z"
Jul 24 21:24:07.589242 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.589207    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4dq9z"
Jul 24 21:24:07.735728 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:07.735629    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4dq9z"]
Jul 24 21:24:07.738345 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:24:07.738317    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd63df2c9_0e80_449d_9fa8_9945622619f6.slice/crio-783fe510285369489ea4215e5489047a9c58ff5e4c4d1c62ac171a7bf871da1b WatchSource:0}: Error finding container 783fe510285369489ea4215e5489047a9c58ff5e4c4d1c62ac171a7bf871da1b: Status 404 returned error can't find the container with id 783fe510285369489ea4215e5489047a9c58ff5e4c4d1c62ac171a7bf871da1b
Jul 24 21:24:08.640721 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:08.640676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4dq9z" event={"ID":"d63df2c9-0e80-449d-9fa8-9945622619f6","Type":"ContainerStarted","Data":"783fe510285369489ea4215e5489047a9c58ff5e4c4d1c62ac171a7bf871da1b"}
Jul 24 21:24:11.656124 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:11.656092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4dq9z" event={"ID":"d63df2c9-0e80-449d-9fa8-9945622619f6","Type":"ContainerStarted","Data":"370d2594fbc616d4a3b08d00281f027abbd06d0bd42cadcf8222c034003ce871"}
Jul 24 21:24:13.665559 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:13.665524    2580 generic.go:356] "Generic (PLEG): container finished" podID="d63df2c9-0e80-449d-9fa8-9945622619f6" containerID="370d2594fbc616d4a3b08d00281f027abbd06d0bd42cadcf8222c034003ce871" exitCode=0
Jul 24 21:24:13.665925 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:13.665580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4dq9z" event={"ID":"d63df2c9-0e80-449d-9fa8-9945622619f6","Type":"ContainerDied","Data":"370d2594fbc616d4a3b08d00281f027abbd06d0bd42cadcf8222c034003ce871"}
Jul 24 21:24:18.693192 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:24:18.693162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4dq9z" event={"ID":"d63df2c9-0e80-449d-9fa8-9945622619f6","Type":"ContainerStarted","Data":"64796c87d65bf5837bc3d123228652a720824443f3b63f97c051cbc3d758c31f"}
Jul 24 21:25:23.961943 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:23.961911    2580 generic.go:356] "Generic (PLEG): container finished" podID="d63df2c9-0e80-449d-9fa8-9945622619f6" containerID="64796c87d65bf5837bc3d123228652a720824443f3b63f97c051cbc3d758c31f" exitCode=0
Jul 24 21:25:23.962340 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:23.961975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4dq9z" event={"ID":"d63df2c9-0e80-449d-9fa8-9945622619f6","Type":"ContainerDied","Data":"64796c87d65bf5837bc3d123228652a720824443f3b63f97c051cbc3d758c31f"}
Jul 24 21:25:25.096513 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.096488    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4dq9z"
Jul 24 21:25:25.104101 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.104083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63df2c9-0e80-449d-9fa8-9945622619f6-model-cache\") pod \"d63df2c9-0e80-449d-9fa8-9945622619f6\" (UID: \"d63df2c9-0e80-449d-9fa8-9945622619f6\") "
Jul 24 21:25:25.104169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.104136    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kkcl\" (UniqueName: \"kubernetes.io/projected/d63df2c9-0e80-449d-9fa8-9945622619f6-kube-api-access-7kkcl\") pod \"d63df2c9-0e80-449d-9fa8-9945622619f6\" (UID: \"d63df2c9-0e80-449d-9fa8-9945622619f6\") "
Jul 24 21:25:25.106455 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.106426    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63df2c9-0e80-449d-9fa8-9945622619f6-kube-api-access-7kkcl" (OuterVolumeSpecName: "kube-api-access-7kkcl") pod "d63df2c9-0e80-449d-9fa8-9945622619f6" (UID: "d63df2c9-0e80-449d-9fa8-9945622619f6"). InnerVolumeSpecName "kube-api-access-7kkcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:25:25.127785 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.127749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63df2c9-0e80-449d-9fa8-9945622619f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "d63df2c9-0e80-449d-9fa8-9945622619f6" (UID: "d63df2c9-0e80-449d-9fa8-9945622619f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:25:25.205045 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.205007    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63df2c9-0e80-449d-9fa8-9945622619f6-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:25:25.205045 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.205035    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kkcl\" (UniqueName: \"kubernetes.io/projected/d63df2c9-0e80-449d-9fa8-9945622619f6-kube-api-access-7kkcl\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:25:25.972656 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.972629    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4dq9z"
Jul 24 21:25:25.972656 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.972649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4dq9z" event={"ID":"d63df2c9-0e80-449d-9fa8-9945622619f6","Type":"ContainerDied","Data":"783fe510285369489ea4215e5489047a9c58ff5e4c4d1c62ac171a7bf871da1b"}
Jul 24 21:25:25.972852 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:25.972680    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="783fe510285369489ea4215e5489047a9c58ff5e4c4d1c62ac171a7bf871da1b"
Jul 24 21:25:26.103591 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:26.103551    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:25:26.115069 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:26.115044    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:25:34.702057 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.702025    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"]
Jul 24 21:25:34.702517 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.702373    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d63df2c9-0e80-449d-9fa8-9945622619f6" containerName="s3-init"
Jul 24 21:25:34.702517 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.702401    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d63df2c9-0e80-449d-9fa8-9945622619f6" containerName="s3-init"
Jul 24 21:25:34.702517 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.702440    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d63df2c9-0e80-449d-9fa8-9945622619f6" containerName="download-hf-model"
Jul 24 21:25:34.702517 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.702446    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d63df2c9-0e80-449d-9fa8-9945622619f6" containerName="download-hf-model"
Jul 24 21:25:34.702517 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.702504    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d63df2c9-0e80-449d-9fa8-9945622619f6" containerName="s3-init"
Jul 24 21:25:34.705515 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.705500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:34.708116 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.708096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 24 21:25:34.708250 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.708124    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 24 21:25:34.714238 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.714217    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"]
Jul 24 21:25:34.796650 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.796615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9d923c4-9c10-4a98-9abd-02fabb376f17-data\") pod \"seaweedfs-tls-custom-8b44d767d-8m2rk\" (UID: \"d9d923c4-9c10-4a98-9abd-02fabb376f17\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:34.796650 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.796652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmxj\" (UniqueName: \"kubernetes.io/projected/d9d923c4-9c10-4a98-9abd-02fabb376f17-kube-api-access-7gmxj\") pod \"seaweedfs-tls-custom-8b44d767d-8m2rk\" (UID: \"d9d923c4-9c10-4a98-9abd-02fabb376f17\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:34.796895 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.796679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/d9d923c4-9c10-4a98-9abd-02fabb376f17-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-8m2rk\" (UID: \"d9d923c4-9c10-4a98-9abd-02fabb376f17\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:34.897259 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.897226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9d923c4-9c10-4a98-9abd-02fabb376f17-data\") pod \"seaweedfs-tls-custom-8b44d767d-8m2rk\" (UID: \"d9d923c4-9c10-4a98-9abd-02fabb376f17\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:34.897259 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.897265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmxj\" (UniqueName: \"kubernetes.io/projected/d9d923c4-9c10-4a98-9abd-02fabb376f17-kube-api-access-7gmxj\") pod \"seaweedfs-tls-custom-8b44d767d-8m2rk\" (UID: \"d9d923c4-9c10-4a98-9abd-02fabb376f17\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:34.897463 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.897291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/d9d923c4-9c10-4a98-9abd-02fabb376f17-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-8m2rk\" (UID: \"d9d923c4-9c10-4a98-9abd-02fabb376f17\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:34.897688 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.897666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9d923c4-9c10-4a98-9abd-02fabb376f17-data\") pod \"seaweedfs-tls-custom-8b44d767d-8m2rk\" (UID: \"d9d923c4-9c10-4a98-9abd-02fabb376f17\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:34.899828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.899801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/d9d923c4-9c10-4a98-9abd-02fabb376f17-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-8m2rk\" (UID: \"d9d923c4-9c10-4a98-9abd-02fabb376f17\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:34.905747 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:34.905725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmxj\" (UniqueName: \"kubernetes.io/projected/d9d923c4-9c10-4a98-9abd-02fabb376f17-kube-api-access-7gmxj\") pod \"seaweedfs-tls-custom-8b44d767d-8m2rk\" (UID: \"d9d923c4-9c10-4a98-9abd-02fabb376f17\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:35.015136 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:35.015115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"
Jul 24 21:25:35.148654 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:35.148624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-8m2rk"]
Jul 24 21:25:35.152552 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:25:35.152523    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d923c4_9c10_4a98_9abd_02fabb376f17.slice/crio-a0dc2a9e76d464534d1426cc6829a1209b2eba058cee6007b2ff4d49532795fe WatchSource:0}: Error finding container a0dc2a9e76d464534d1426cc6829a1209b2eba058cee6007b2ff4d49532795fe: Status 404 returned error can't find the container with id a0dc2a9e76d464534d1426cc6829a1209b2eba058cee6007b2ff4d49532795fe
Jul 24 21:25:36.017241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.017202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk" event={"ID":"d9d923c4-9c10-4a98-9abd-02fabb376f17","Type":"ContainerStarted","Data":"402118ab36c756c68c3623fa7bb2eedf0c8bd97131aa2b595badc5fb808e5996"}
Jul 24 21:25:36.017241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.017246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk" event={"ID":"d9d923c4-9c10-4a98-9abd-02fabb376f17","Type":"ContainerStarted","Data":"a0dc2a9e76d464534d1426cc6829a1209b2eba058cee6007b2ff4d49532795fe"}
Jul 24 21:25:36.033742 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.033694    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-8m2rk" podStartSLOduration=1.777475521 podStartE2EDuration="2.033677859s" totalImagesPullingTime="256.202338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:25:36.03316142 +0000 UTC m=+610.372003197" watchObservedRunningTime="2026-07-24 21:25:36.033677859 +0000 UTC m=+610.372519634"
Jul 24 21:25:36.339574 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.339490    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-sbgpg"]
Jul 24 21:25:36.343947 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.343930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:36.350597 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.350577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-sbgpg"]
Jul 24 21:25:36.411509 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.411473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df498cd-676f-407e-9833-68d40b1656a9-model-cache\") pod \"s3-tls-init-custom-sbgpg\" (UID: \"5df498cd-676f-407e-9833-68d40b1656a9\") " pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:36.411696 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.411521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tls9p\" (UniqueName: \"kubernetes.io/projected/5df498cd-676f-407e-9833-68d40b1656a9-kube-api-access-tls9p\") pod \"s3-tls-init-custom-sbgpg\" (UID: \"5df498cd-676f-407e-9833-68d40b1656a9\") " pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:36.512480 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.512446    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df498cd-676f-407e-9833-68d40b1656a9-model-cache\") pod \"s3-tls-init-custom-sbgpg\" (UID: \"5df498cd-676f-407e-9833-68d40b1656a9\") " pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:36.512480 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.512485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tls9p\" (UniqueName: \"kubernetes.io/projected/5df498cd-676f-407e-9833-68d40b1656a9-kube-api-access-tls9p\") pod \"s3-tls-init-custom-sbgpg\" (UID: \"5df498cd-676f-407e-9833-68d40b1656a9\") " pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:36.512822 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.512804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df498cd-676f-407e-9833-68d40b1656a9-model-cache\") pod \"s3-tls-init-custom-sbgpg\" (UID: \"5df498cd-676f-407e-9833-68d40b1656a9\") " pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:36.521357 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.521331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tls9p\" (UniqueName: \"kubernetes.io/projected/5df498cd-676f-407e-9833-68d40b1656a9-kube-api-access-tls9p\") pod \"s3-tls-init-custom-sbgpg\" (UID: \"5df498cd-676f-407e-9833-68d40b1656a9\") " pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:36.660962 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.660882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:36.789122 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:36.789007    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-sbgpg"]
Jul 24 21:25:36.791811 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:25:36.791779    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5df498cd_676f_407e_9833_68d40b1656a9.slice/crio-4c15479899df01b96a1a8cc854ba0cdda271f88c321e5967a255a61450c9b7bc WatchSource:0}: Error finding container 4c15479899df01b96a1a8cc854ba0cdda271f88c321e5967a255a61450c9b7bc: Status 404 returned error can't find the container with id 4c15479899df01b96a1a8cc854ba0cdda271f88c321e5967a255a61450c9b7bc
Jul 24 21:25:37.023538 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:37.023499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sbgpg" event={"ID":"5df498cd-676f-407e-9833-68d40b1656a9","Type":"ContainerStarted","Data":"0d92f04899de6c0b2955e46105a16f7f99d4e86571545a71c7dfc624260f4d5a"}
Jul 24 21:25:37.024026 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:37.023557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sbgpg" event={"ID":"5df498cd-676f-407e-9833-68d40b1656a9","Type":"ContainerStarted","Data":"4c15479899df01b96a1a8cc854ba0cdda271f88c321e5967a255a61450c9b7bc"}
Jul 24 21:25:39.034368 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:39.034332    2580 generic.go:356] "Generic (PLEG): container finished" podID="5df498cd-676f-407e-9833-68d40b1656a9" containerID="0d92f04899de6c0b2955e46105a16f7f99d4e86571545a71c7dfc624260f4d5a" exitCode=0
Jul 24 21:25:39.034758 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:39.034436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sbgpg" event={"ID":"5df498cd-676f-407e-9833-68d40b1656a9","Type":"ContainerDied","Data":"0d92f04899de6c0b2955e46105a16f7f99d4e86571545a71c7dfc624260f4d5a"}
Jul 24 21:25:40.040463 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:40.040426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sbgpg" event={"ID":"5df498cd-676f-407e-9833-68d40b1656a9","Type":"ContainerStarted","Data":"3f8b2dd5f527c58b5ffca14f8a3dd51332f7758dbb4f3098b9400c2d3e6d1ce0"}
Jul 24 21:25:40.057196 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:40.057147    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-sbgpg" podStartSLOduration=2.273375698 podStartE2EDuration="4.05713165s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.783755952s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:25:40.056189566 +0000 UTC m=+614.395031342" watchObservedRunningTime="2026-07-24 21:25:40.05713165 +0000 UTC m=+614.395973425"
Jul 24 21:25:46.068504 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:46.068468    2580 generic.go:356] "Generic (PLEG): container finished" podID="5df498cd-676f-407e-9833-68d40b1656a9" containerID="3f8b2dd5f527c58b5ffca14f8a3dd51332f7758dbb4f3098b9400c2d3e6d1ce0" exitCode=0
Jul 24 21:25:46.068973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:46.068551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sbgpg" event={"ID":"5df498cd-676f-407e-9833-68d40b1656a9","Type":"ContainerDied","Data":"3f8b2dd5f527c58b5ffca14f8a3dd51332f7758dbb4f3098b9400c2d3e6d1ce0"}
Jul 24 21:25:47.228343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:47.228316    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:47.319709 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:47.319679    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tls9p\" (UniqueName: \"kubernetes.io/projected/5df498cd-676f-407e-9833-68d40b1656a9-kube-api-access-tls9p\") pod \"5df498cd-676f-407e-9833-68d40b1656a9\" (UID: \"5df498cd-676f-407e-9833-68d40b1656a9\") "
Jul 24 21:25:47.319902 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:47.319738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df498cd-676f-407e-9833-68d40b1656a9-model-cache\") pod \"5df498cd-676f-407e-9833-68d40b1656a9\" (UID: \"5df498cd-676f-407e-9833-68d40b1656a9\") "
Jul 24 21:25:47.321934 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:47.321907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df498cd-676f-407e-9833-68d40b1656a9-kube-api-access-tls9p" (OuterVolumeSpecName: "kube-api-access-tls9p") pod "5df498cd-676f-407e-9833-68d40b1656a9" (UID: "5df498cd-676f-407e-9833-68d40b1656a9"). InnerVolumeSpecName "kube-api-access-tls9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:25:47.341675 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:47.341648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df498cd-676f-407e-9833-68d40b1656a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5df498cd-676f-407e-9833-68d40b1656a9" (UID: "5df498cd-676f-407e-9833-68d40b1656a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:25:47.421248 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:47.421217    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tls9p\" (UniqueName: \"kubernetes.io/projected/5df498cd-676f-407e-9833-68d40b1656a9-kube-api-access-tls9p\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:25:47.421248 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:47.421242    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df498cd-676f-407e-9833-68d40b1656a9-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:25:48.079670 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:48.079647    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-sbgpg"
Jul 24 21:25:48.079670 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:48.079658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-sbgpg" event={"ID":"5df498cd-676f-407e-9833-68d40b1656a9","Type":"ContainerDied","Data":"4c15479899df01b96a1a8cc854ba0cdda271f88c321e5967a255a61450c9b7bc"}
Jul 24 21:25:48.079863 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:48.079693    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c15479899df01b96a1a8cc854ba0cdda271f88c321e5967a255a61450c9b7bc"
Jul 24 21:25:53.562215 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.562183    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-4tcvm"]
Jul 24 21:25:53.562628 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.562543    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df498cd-676f-407e-9833-68d40b1656a9" containerName="download-hf-model"
Jul 24 21:25:53.562628 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.562554    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df498cd-676f-407e-9833-68d40b1656a9" containerName="download-hf-model"
Jul 24 21:25:53.562628 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.562575    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df498cd-676f-407e-9833-68d40b1656a9" containerName="s3-tls-init-custom"
Jul 24 21:25:53.562628 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.562583    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df498cd-676f-407e-9833-68d40b1656a9" containerName="s3-tls-init-custom"
Jul 24 21:25:53.562770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.562651    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5df498cd-676f-407e-9833-68d40b1656a9" containerName="s3-tls-init-custom"
Jul 24 21:25:53.565098 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.565081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:25:53.567700 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.567686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 24 21:25:53.575772 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.575750    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4tcvm"]
Jul 24 21:25:53.680505 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.680475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l22t2\" (UniqueName: \"kubernetes.io/projected/8dfff6e5-ea64-4a5e-a317-afb411385378-kube-api-access-l22t2\") pod \"s3-tls-init-serving-4tcvm\" (UID: \"8dfff6e5-ea64-4a5e-a317-afb411385378\") " pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:25:53.680505 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.680509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfff6e5-ea64-4a5e-a317-afb411385378-model-cache\") pod \"s3-tls-init-serving-4tcvm\" (UID: \"8dfff6e5-ea64-4a5e-a317-afb411385378\") " pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:25:53.781326 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.781295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22t2\" (UniqueName: \"kubernetes.io/projected/8dfff6e5-ea64-4a5e-a317-afb411385378-kube-api-access-l22t2\") pod \"s3-tls-init-serving-4tcvm\" (UID: \"8dfff6e5-ea64-4a5e-a317-afb411385378\") " pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:25:53.781326 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.781330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfff6e5-ea64-4a5e-a317-afb411385378-model-cache\") pod \"s3-tls-init-serving-4tcvm\" (UID: \"8dfff6e5-ea64-4a5e-a317-afb411385378\") " pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:25:53.781672 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.781655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfff6e5-ea64-4a5e-a317-afb411385378-model-cache\") pod \"s3-tls-init-serving-4tcvm\" (UID: \"8dfff6e5-ea64-4a5e-a317-afb411385378\") " pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:25:53.791016 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.790987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l22t2\" (UniqueName: \"kubernetes.io/projected/8dfff6e5-ea64-4a5e-a317-afb411385378-kube-api-access-l22t2\") pod \"s3-tls-init-serving-4tcvm\" (UID: \"8dfff6e5-ea64-4a5e-a317-afb411385378\") " pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:25:53.881947 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:53.881859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:25:54.018237 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:54.018119    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-4tcvm"]
Jul 24 21:25:54.020573 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:25:54.020535    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dfff6e5_ea64_4a5e_a317_afb411385378.slice/crio-e4e95f7ef591806921532daaed57494b951cca778e0e58801332890f5878eaf9 WatchSource:0}: Error finding container e4e95f7ef591806921532daaed57494b951cca778e0e58801332890f5878eaf9: Status 404 returned error can't find the container with id e4e95f7ef591806921532daaed57494b951cca778e0e58801332890f5878eaf9
Jul 24 21:25:54.107012 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:54.106982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4tcvm" event={"ID":"8dfff6e5-ea64-4a5e-a317-afb411385378","Type":"ContainerStarted","Data":"bad1a242d788632bc6af0ad5b7ece71bcca4b1bcdb69578a9b521698422e7e53"}
Jul 24 21:25:54.107150 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:54.107019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4tcvm" event={"ID":"8dfff6e5-ea64-4a5e-a317-afb411385378","Type":"ContainerStarted","Data":"e4e95f7ef591806921532daaed57494b951cca778e0e58801332890f5878eaf9"}
Jul 24 21:25:57.123095 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:57.123057    2580 generic.go:356] "Generic (PLEG): container finished" podID="8dfff6e5-ea64-4a5e-a317-afb411385378" containerID="bad1a242d788632bc6af0ad5b7ece71bcca4b1bcdb69578a9b521698422e7e53" exitCode=0
Jul 24 21:25:57.123564 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:57.123110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4tcvm" event={"ID":"8dfff6e5-ea64-4a5e-a317-afb411385378","Type":"ContainerDied","Data":"bad1a242d788632bc6af0ad5b7ece71bcca4b1bcdb69578a9b521698422e7e53"}
Jul 24 21:25:58.129355 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:58.129322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4tcvm" event={"ID":"8dfff6e5-ea64-4a5e-a317-afb411385378","Type":"ContainerStarted","Data":"7fb6a0d6bd2e331237c12533ca214742bafe5776f749d3cacf2a405a180ee724"}
Jul 24 21:25:58.147727 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:25:58.147655    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-4tcvm" podStartSLOduration=2.979265719 podStartE2EDuration="5.147635084s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.168369365s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:25:58.146806223 +0000 UTC m=+632.485648000" watchObservedRunningTime="2026-07-24 21:25:58.147635084 +0000 UTC m=+632.486476860"
Jul 24 21:26:03.156086 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:03.156054    2580 generic.go:356] "Generic (PLEG): container finished" podID="8dfff6e5-ea64-4a5e-a317-afb411385378" containerID="7fb6a0d6bd2e331237c12533ca214742bafe5776f749d3cacf2a405a180ee724" exitCode=0
Jul 24 21:26:03.156558 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:03.156130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4tcvm" event={"ID":"8dfff6e5-ea64-4a5e-a317-afb411385378","Type":"ContainerDied","Data":"7fb6a0d6bd2e331237c12533ca214742bafe5776f749d3cacf2a405a180ee724"}
Jul 24 21:26:04.329139 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:04.329116    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:26:04.376860 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:04.376828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l22t2\" (UniqueName: \"kubernetes.io/projected/8dfff6e5-ea64-4a5e-a317-afb411385378-kube-api-access-l22t2\") pod \"8dfff6e5-ea64-4a5e-a317-afb411385378\" (UID: \"8dfff6e5-ea64-4a5e-a317-afb411385378\") "
Jul 24 21:26:04.377017 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:04.376889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfff6e5-ea64-4a5e-a317-afb411385378-model-cache\") pod \"8dfff6e5-ea64-4a5e-a317-afb411385378\" (UID: \"8dfff6e5-ea64-4a5e-a317-afb411385378\") "
Jul 24 21:26:04.379058 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:04.379027    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfff6e5-ea64-4a5e-a317-afb411385378-kube-api-access-l22t2" (OuterVolumeSpecName: "kube-api-access-l22t2") pod "8dfff6e5-ea64-4a5e-a317-afb411385378" (UID: "8dfff6e5-ea64-4a5e-a317-afb411385378"). InnerVolumeSpecName "kube-api-access-l22t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:26:04.399162 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:04.399135    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfff6e5-ea64-4a5e-a317-afb411385378-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dfff6e5-ea64-4a5e-a317-afb411385378" (UID: "8dfff6e5-ea64-4a5e-a317-afb411385378"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:26:04.477706 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:04.477628    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l22t2\" (UniqueName: \"kubernetes.io/projected/8dfff6e5-ea64-4a5e-a317-afb411385378-kube-api-access-l22t2\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:26:04.477706 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:04.477652    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dfff6e5-ea64-4a5e-a317-afb411385378-model-cache\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:26:05.166361 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:05.166324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-4tcvm" event={"ID":"8dfff6e5-ea64-4a5e-a317-afb411385378","Type":"ContainerDied","Data":"e4e95f7ef591806921532daaed57494b951cca778e0e58801332890f5878eaf9"}
Jul 24 21:26:05.166361 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:05.166348    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-4tcvm"
Jul 24 21:26:05.166361 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:05.166369    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4e95f7ef591806921532daaed57494b951cca778e0e58801332890f5878eaf9"
Jul 24 21:26:17.908316 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.908280    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"]
Jul 24 21:26:17.909948 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.908807    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfff6e5-ea64-4a5e-a317-afb411385378" containerName="s3-tls-init-serving"
Jul 24 21:26:17.909948 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.908822    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfff6e5-ea64-4a5e-a317-afb411385378" containerName="s3-tls-init-serving"
Jul 24 21:26:17.909948 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.908841    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfff6e5-ea64-4a5e-a317-afb411385378" containerName="download-hf-model"
Jul 24 21:26:17.909948 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.908846    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfff6e5-ea64-4a5e-a317-afb411385378" containerName="download-hf-model"
Jul 24 21:26:17.909948 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.908908    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dfff6e5-ea64-4a5e-a317-afb411385378" containerName="s3-tls-init-serving"
Jul 24 21:26:17.911193 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.911174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:17.913546 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.913526    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:26:17.913658 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.913587    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:26:17.913658 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.913630    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:26:17.913767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.913700    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 24 21:26:17.913767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.913707    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:26:17.913767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.913746    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:26:17.913917 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.913748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 24 21:26:17.923252 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.923229    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"]
Jul 24 21:26:17.997117 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.997070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:17.997286 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.997136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:17.997286 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.997199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:17.997286 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.997223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:17.997286 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:17.997247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7php\" (UniqueName: \"kubernetes.io/projected/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-api-access-d7php\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.098155 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.098123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.098331 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.098179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.098331 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.098209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.098331 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.098223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.098331 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.098248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7php\" (UniqueName: \"kubernetes.io/projected/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-api-access-d7php\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.098667 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.098645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.098954 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.098934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.098954 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.098948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.100664 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.100645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.105852 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.105833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7php\" (UniqueName: \"kubernetes.io/projected/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-api-access-d7php\") pod \"isvc-sklearn-batcher-predictor-7776d964d7-w2t6x\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.221793 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.221704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:18.358241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:18.358212    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"]
Jul 24 21:26:18.360747 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:26:18.360721    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787ae0a4_34c8_4a8a_b9ee_9014f3ef5017.slice/crio-bc5a8ad07eb8cae4ee56e648afa422714a51bd22381bd9b4fea41fd4f30915f2 WatchSource:0}: Error finding container bc5a8ad07eb8cae4ee56e648afa422714a51bd22381bd9b4fea41fd4f30915f2: Status 404 returned error can't find the container with id bc5a8ad07eb8cae4ee56e648afa422714a51bd22381bd9b4fea41fd4f30915f2
Jul 24 21:26:19.227664 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:19.227630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerStarted","Data":"05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497"}
Jul 24 21:26:19.228131 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:19.227708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerStarted","Data":"bc5a8ad07eb8cae4ee56e648afa422714a51bd22381bd9b4fea41fd4f30915f2"}
Jul 24 21:26:23.247410 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:23.247361    2580 generic.go:356] "Generic (PLEG): container finished" podID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerID="05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497" exitCode=0
Jul 24 21:26:23.247836 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:23.247434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerDied","Data":"05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497"}
Jul 24 21:26:36.327325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:36.327239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerStarted","Data":"22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162"}
Jul 24 21:26:39.345421 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:39.345370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerStarted","Data":"49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094"}
Jul 24 21:26:41.356567 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:41.356532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerStarted","Data":"5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2"}
Jul 24 21:26:41.356957 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:41.356648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:41.377939 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:41.377890    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podStartSLOduration=3.36410447 podStartE2EDuration="24.377874984s" totalImagesPullingTime="17.108178269s" totalInitContainerRuntime="3.905592245s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:26:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:26:41.37651394 +0000 UTC m=+675.715355727" watchObservedRunningTime="2026-07-24 21:26:41.377874984 +0000 UTC m=+675.716716759"
Jul 24 21:26:42.361905 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:42.361869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:42.362263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:42.361943    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:42.363143 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:42.363105    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:26:42.363746 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:42.363725    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:43.367047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:43.366993    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:26:43.367477 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:43.367430    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:43.370857 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:43.370835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:26:44.371851 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:44.371811    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:26:44.372254 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:44.372077    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:54.372647 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:54.372607    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:26:54.373179 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:26:54.373153    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:04.372328 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:04.372279    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:27:04.372814 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:04.372748    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:14.372270 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:14.372223    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:27:14.372723 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:14.372701    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:24.371810 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:24.371719    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:27:24.372247 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:24.372153    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:34.371969 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:34.371922    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:27:34.372522 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:34.372499    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:44.372191 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:44.372140    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:27:44.372720 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:44.372621    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:45.282649 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:45.282602    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:27:45.282907 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:45.282883    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:55.283518 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:55.283485    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:27:55.284018 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:27:55.283575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:28:03.136945 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.136916    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"]
Jul 24 21:28:03.137530 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.137182    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" containerID="cri-o://22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162" gracePeriod=30
Jul 24 21:28:03.137530 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.137217    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" containerID="cri-o://5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2" gracePeriod=30
Jul 24 21:28:03.137530 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.137212    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy" containerID="cri-o://49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094" gracePeriod=30
Jul 24 21:28:03.173301 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.173268    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"]
Jul 24 21:28:03.176532 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.176507    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.179076 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.179051    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 24 21:28:03.179199 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.179055    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 24 21:28:03.188271 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.188247    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"]
Jul 24 21:28:03.249443 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.249379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.249582 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.249478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30dae70-b697-41b6-bda3-0319d839261a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.249582 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.249498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f30dae70-b697-41b6-bda3-0319d839261a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.249582 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.249542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.249582 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.249574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrqz\" (UniqueName: \"kubernetes.io/projected/f30dae70-b697-41b6-bda3-0319d839261a-kube-api-access-7lrqz\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.350703 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.350666    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30dae70-b697-41b6-bda3-0319d839261a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.350703 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.350707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f30dae70-b697-41b6-bda3-0319d839261a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.350979 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.350765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.350979 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.350819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrqz\" (UniqueName: \"kubernetes.io/projected/f30dae70-b697-41b6-bda3-0319d839261a-kube-api-access-7lrqz\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.350979 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:28:03.350846    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 24 21:28:03.350979 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.350869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.350979 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:28:03.350913    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f30dae70-b697-41b6-bda3-0319d839261a-proxy-tls podName:f30dae70-b697-41b6-bda3-0319d839261a nodeName:}" failed. No retries permitted until 2026-07-24 21:28:03.850891936 +0000 UTC m=+758.189733762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f30dae70-b697-41b6-bda3-0319d839261a-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" (UID: "f30dae70-b697-41b6-bda3-0319d839261a") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 24 21:28:03.351218 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.351117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30dae70-b697-41b6-bda3-0319d839261a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.351552 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.351533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.351699 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.351682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.360914 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.360891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrqz\" (UniqueName: \"kubernetes.io/projected/f30dae70-b697-41b6-bda3-0319d839261a-kube-api-access-7lrqz\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.367967 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.367936    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 24 21:28:03.757251 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.757220    2580 generic.go:356] "Generic (PLEG): container finished" podID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerID="49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094" exitCode=2
Jul 24 21:28:03.757448 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.757280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerDied","Data":"49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094"}
Jul 24 21:28:03.855700 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.855667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f30dae70-b697-41b6-bda3-0319d839261a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:03.858200 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:03.858172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f30dae70-b697-41b6-bda3-0319d839261a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:04.086667 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:04.086623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:04.224312 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:04.224286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"]
Jul 24 21:28:04.227019 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:28:04.226993    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30dae70_b697_41b6_bda3_0319d839261a.slice/crio-bf1e7ec6ba85ed0fe2327fe4f4febf52fe9a31b50e70925616eeeb60fdc8d57c WatchSource:0}: Error finding container bf1e7ec6ba85ed0fe2327fe4f4febf52fe9a31b50e70925616eeeb60fdc8d57c: Status 404 returned error can't find the container with id bf1e7ec6ba85ed0fe2327fe4f4febf52fe9a31b50e70925616eeeb60fdc8d57c
Jul 24 21:28:04.764262 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:04.764220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerStarted","Data":"79c75406ee8f4a2d4de668671d52574ea43cc6faff55250f927c7247c91cff3c"}
Jul 24 21:28:04.764509 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:04.764287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerStarted","Data":"bf1e7ec6ba85ed0fe2327fe4f4febf52fe9a31b50e70925616eeeb60fdc8d57c"}
Jul 24 21:28:05.282729 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:05.282679    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:28:05.283189 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:05.283009    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:07.781317 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:07.781287    2580 generic.go:356] "Generic (PLEG): container finished" podID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerID="22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162" exitCode=0
Jul 24 21:28:07.781686 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:07.781343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerDied","Data":"22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162"}
Jul 24 21:28:08.368087 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:08.368041    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 24 21:28:08.787871 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:08.787836    2580 generic.go:356] "Generic (PLEG): container finished" podID="f30dae70-b697-41b6-bda3-0319d839261a" containerID="79c75406ee8f4a2d4de668671d52574ea43cc6faff55250f927c7247c91cff3c" exitCode=0
Jul 24 21:28:08.788245 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:08.787911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerDied","Data":"79c75406ee8f4a2d4de668671d52574ea43cc6faff55250f927c7247c91cff3c"}
Jul 24 21:28:09.795862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:09.795829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerStarted","Data":"b34c985e853a4b8d461899c20f6ddbaac0dc88e28db4f13ba2fdc7c84d93aa9f"}
Jul 24 21:28:09.796260 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:09.795897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerStarted","Data":"1ec2adabaef1d7687e6680448fd9f488b34d6cd51446400f4ad921ce0b266fbc"}
Jul 24 21:28:09.796260 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:09.795921    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerStarted","Data":"f66d8060a00b6f415ddb98569119bef77f100fa8f3508d39b4198db28ad07498"}
Jul 24 21:28:09.796260 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:09.796196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:09.796384 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:09.796337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:09.797731 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:09.797703    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:28:09.819048 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:09.819008    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podStartSLOduration=2.915329232 podStartE2EDuration="6.818995437s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.903666205s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:28:09.816528567 +0000 UTC m=+764.155370387" watchObservedRunningTime="2026-07-24 21:28:09.818995437 +0000 UTC m=+764.157837213"
Jul 24 21:28:10.801142 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:10.801110    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:10.801571 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:10.801170    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:28:10.802430 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:10.802382    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:11.805904 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:11.805863    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:28:11.806336 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:11.806204    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:13.368036 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:13.367997    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 24 21:28:13.368455 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:13.368133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:28:15.282657 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:15.282614    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:28:15.283122 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:15.282933    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:16.810325 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:16.810298    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:28:16.810904 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:16.810859    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:28:16.811337 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:16.811313    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:18.367576 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:18.367538    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 24 21:28:23.367773 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:23.367730    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 24 21:28:25.282575 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:25.282529    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:28:25.283005 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:25.282691    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:28:25.283005 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:25.282797    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:25.283005 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:25.282921    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:28:26.811159 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:26.811111    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:28:26.811601 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:26.811572    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:28.367989 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:28.367952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 24 21:28:33.298863 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.298836    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:28:33.423264 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423185    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-rbac-proxy-sar-config\") pod \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") "
Jul 24 21:28:33.423264 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423248    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7php\" (UniqueName: \"kubernetes.io/projected/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-api-access-d7php\") pod \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") "
Jul 24 21:28:33.423475 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423281    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kserve-provision-location\") pod \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") "
Jul 24 21:28:33.423475 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-cabundle-cert\") pod \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") "
Jul 24 21:28:33.423475 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-proxy-tls\") pod \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\" (UID: \"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017\") "
Jul 24 21:28:33.423713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" (UID: "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:28:33.423713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423655    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" (UID: "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:28:33.423713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" (UID: "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:28:33.423945 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423765    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:28:33.423945 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423779    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:28:33.423945 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.423788    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:28:33.425654 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.425618    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" (UID: "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:28:33.425654 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.425637    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-api-access-d7php" (OuterVolumeSpecName: "kube-api-access-d7php") pod "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" (UID: "787ae0a4-34c8-4a8a-b9ee-9014f3ef5017"). InnerVolumeSpecName "kube-api-access-d7php". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:28:33.524756 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.524721    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:28:33.524756 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.524751    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7php\" (UniqueName: \"kubernetes.io/projected/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017-kube-api-access-d7php\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:28:33.916721 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.916685    2580 generic.go:356] "Generic (PLEG): container finished" podID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerID="5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2" exitCode=0
Jul 24 21:28:33.916863 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.916757    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"
Jul 24 21:28:33.916863 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.916766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerDied","Data":"5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2"}
Jul 24 21:28:33.916863 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.916856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x" event={"ID":"787ae0a4-34c8-4a8a-b9ee-9014f3ef5017","Type":"ContainerDied","Data":"bc5a8ad07eb8cae4ee56e648afa422714a51bd22381bd9b4fea41fd4f30915f2"}
Jul 24 21:28:33.916984 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.916889    2580 scope.go:117] "RemoveContainer" containerID="5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2"
Jul 24 21:28:33.926067 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.926048    2580 scope.go:117] "RemoveContainer" containerID="49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094"
Jul 24 21:28:33.933737 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.933718    2580 scope.go:117] "RemoveContainer" containerID="22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162"
Jul 24 21:28:33.940015 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.939994    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"]
Jul 24 21:28:33.941928 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.941899    2580 scope.go:117] "RemoveContainer" containerID="05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497"
Jul 24 21:28:33.944092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.944072    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7776d964d7-w2t6x"]
Jul 24 21:28:33.949607 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.949589    2580 scope.go:117] "RemoveContainer" containerID="5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2"
Jul 24 21:28:33.949871 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:28:33.949855    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2\": container with ID starting with 5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2 not found: ID does not exist" containerID="5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2"
Jul 24 21:28:33.949928 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.949887    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2"} err="failed to get container status \"5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2\": rpc error: code = NotFound desc = could not find container \"5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2\": container with ID starting with 5fd2275c65d7f4f2dbaf3670ed8b5978ed96ccc325ccdcd5d9933c1c1fec85f2 not found: ID does not exist"
Jul 24 21:28:33.949928 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.949915    2580 scope.go:117] "RemoveContainer" containerID="49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094"
Jul 24 21:28:33.950146 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:28:33.950129    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094\": container with ID starting with 49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094 not found: ID does not exist" containerID="49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094"
Jul 24 21:28:33.950185 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.950152    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094"} err="failed to get container status \"49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094\": rpc error: code = NotFound desc = could not find container \"49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094\": container with ID starting with 49cb2ac987a9e8fb00addae9bce1145ad0b626abad528a32740ca4fffdf2a094 not found: ID does not exist"
Jul 24 21:28:33.950185 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.950168    2580 scope.go:117] "RemoveContainer" containerID="22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162"
Jul 24 21:28:33.950402 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:28:33.950373    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162\": container with ID starting with 22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162 not found: ID does not exist" containerID="22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162"
Jul 24 21:28:33.950447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.950415    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162"} err="failed to get container status \"22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162\": rpc error: code = NotFound desc = could not find container \"22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162\": container with ID starting with 22667ad02c875763f70d9014c5f8d3fd99558c9707d988510a25d19c29ffb162 not found: ID does not exist"
Jul 24 21:28:33.950447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.950430    2580 scope.go:117] "RemoveContainer" containerID="05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497"
Jul 24 21:28:33.950602 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:28:33.950585    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497\": container with ID starting with 05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497 not found: ID does not exist" containerID="05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497"
Jul 24 21:28:33.950653 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:33.950606    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497"} err="failed to get container status \"05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497\": rpc error: code = NotFound desc = could not find container \"05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497\": container with ID starting with 05512994f36ffd8e267df415c7d5d0c24c8d5ede14697b9127dd591c330e5497 not found: ID does not exist"
Jul 24 21:28:34.287646 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:34.287617    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" path="/var/lib/kubelet/pods/787ae0a4-34c8-4a8a-b9ee-9014f3ef5017/volumes"
Jul 24 21:28:36.810852 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:36.810801    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:28:36.811310 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:36.811250    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:46.811043 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:46.810950    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:28:46.811550 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:46.811405    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:56.811066 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:56.811026    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:28:56.811605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:28:56.811521    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:29:06.811301 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:06.811257    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:29:06.811803 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:06.811777    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:29:16.810985 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:16.810939    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:29:16.811512 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:16.811455    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:29:26.812289 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:26.812263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:29:26.812758 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:26.812734    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:29:38.330963 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.330932    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"]
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331292    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="storage-initializer"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331304    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="storage-initializer"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331318    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331324    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331340    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331346    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331358    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331363    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331434    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kube-rbac-proxy"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331444    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="kserve-container"
Jul 24 21:29:38.331543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.331453    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="787ae0a4-34c8-4a8a-b9ee-9014f3ef5017" containerName="agent"
Jul 24 21:29:38.335885 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.335866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:38.339143 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.339118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 24 21:29:38.339558 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.339538    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 24 21:29:38.346151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.346125    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"]
Jul 24 21:29:38.357263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.357231    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"]
Jul 24 21:29:38.357553 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.357515    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" containerID="cri-o://f66d8060a00b6f415ddb98569119bef77f100fa8f3508d39b4198db28ad07498" gracePeriod=30
Jul 24 21:29:38.357642 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.357599    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" containerID="cri-o://b34c985e853a4b8d461899c20f6ddbaac0dc88e28db4f13ba2fdc7c84d93aa9f" gracePeriod=30
Jul 24 21:29:38.357704 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.357611    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy" containerID="cri-o://1ec2adabaef1d7687e6680448fd9f488b34d6cd51446400f4ad921ce0b266fbc" gracePeriod=30
Jul 24 21:29:38.398354 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.398320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zk6\" (UniqueName: \"kubernetes.io/projected/27039753-bd28-4b3f-bed8-18259c111ae2-kube-api-access-k8zk6\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:38.398538 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.398411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27039753-bd28-4b3f-bed8-18259c111ae2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:38.398538 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.398443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27039753-bd28-4b3f-bed8-18259c111ae2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:38.499550 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.499503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27039753-bd28-4b3f-bed8-18259c111ae2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:38.499550 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.499551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27039753-bd28-4b3f-bed8-18259c111ae2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:38.499770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.499602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zk6\" (UniqueName: \"kubernetes.io/projected/27039753-bd28-4b3f-bed8-18259c111ae2-kube-api-access-k8zk6\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:38.499770 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:29:38.499740    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 24 21:29:38.499855 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:29:38.499808    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27039753-bd28-4b3f-bed8-18259c111ae2-proxy-tls podName:27039753-bd28-4b3f-bed8-18259c111ae2 nodeName:}" failed. No retries permitted until 2026-07-24 21:29:38.999787077 +0000 UTC m=+853.338628840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/27039753-bd28-4b3f-bed8-18259c111ae2-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-pfc7l" (UID: "27039753-bd28-4b3f-bed8-18259c111ae2") : secret "message-dumper-predictor-serving-cert" not found
Jul 24 21:29:38.500219 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.500199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27039753-bd28-4b3f-bed8-18259c111ae2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:38.508113 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:38.508077    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zk6\" (UniqueName: \"kubernetes.io/projected/27039753-bd28-4b3f-bed8-18259c111ae2-kube-api-access-k8zk6\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:39.004043 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:39.004002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27039753-bd28-4b3f-bed8-18259c111ae2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:39.006569 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:39.006547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27039753-bd28-4b3f-bed8-18259c111ae2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-pfc7l\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:39.226853 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:39.226813    2580 generic.go:356] "Generic (PLEG): container finished" podID="f30dae70-b697-41b6-bda3-0319d839261a" containerID="1ec2adabaef1d7687e6680448fd9f488b34d6cd51446400f4ad921ce0b266fbc" exitCode=2
Jul 24 21:29:39.227016 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:39.226879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerDied","Data":"1ec2adabaef1d7687e6680448fd9f488b34d6cd51446400f4ad921ce0b266fbc"}
Jul 24 21:29:39.247049 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:39.247020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:39.390022 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:39.389987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"]
Jul 24 21:29:39.393878 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:29:39.393853    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27039753_bd28_4b3f_bed8_18259c111ae2.slice/crio-e4948b05cb22bed157865830f0dd111bca0d071af0f96b2ac8f218e64b8e3d8c WatchSource:0}: Error finding container e4948b05cb22bed157865830f0dd111bca0d071af0f96b2ac8f218e64b8e3d8c: Status 404 returned error can't find the container with id e4948b05cb22bed157865830f0dd111bca0d071af0f96b2ac8f218e64b8e3d8c
Jul 24 21:29:39.395791 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:39.395775    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:29:40.233045 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:40.233011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l" event={"ID":"27039753-bd28-4b3f-bed8-18259c111ae2","Type":"ContainerStarted","Data":"e4948b05cb22bed157865830f0dd111bca0d071af0f96b2ac8f218e64b8e3d8c"}
Jul 24 21:29:41.239333 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:41.239296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l" event={"ID":"27039753-bd28-4b3f-bed8-18259c111ae2","Type":"ContainerStarted","Data":"c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485"}
Jul 24 21:29:41.239710 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:41.239354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l" event={"ID":"27039753-bd28-4b3f-bed8-18259c111ae2","Type":"ContainerStarted","Data":"021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc"}
Jul 24 21:29:41.239710 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:41.239442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:41.257618 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:41.257571    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l" podStartSLOduration=1.974443755 podStartE2EDuration="3.257555184s" totalImagesPullingTime="1.283111429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:29:41.256299166 +0000 UTC m=+855.595140943" watchObservedRunningTime="2026-07-24 21:29:41.257555184 +0000 UTC m=+855.596396960"
Jul 24 21:29:41.806363 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:41.806318    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 24 21:29:42.244695 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:42.244617    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:42.246218 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:42.246197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:43.252060 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:43.252025    2580 generic.go:356] "Generic (PLEG): container finished" podID="f30dae70-b697-41b6-bda3-0319d839261a" containerID="f66d8060a00b6f415ddb98569119bef77f100fa8f3508d39b4198db28ad07498" exitCode=0
Jul 24 21:29:43.252470 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:43.252098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerDied","Data":"f66d8060a00b6f415ddb98569119bef77f100fa8f3508d39b4198db28ad07498"}
Jul 24 21:29:46.806130 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:46.806088    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 24 21:29:46.811850 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:46.811823    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:29:46.813054 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:46.813034    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:29:49.262687 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:49.262659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:29:51.805994 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:51.805953    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 24 21:29:51.806376 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:51.806076    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:29:56.806211 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:56.806163    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 24 21:29:56.811890 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:56.811867    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:29:56.813058 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:56.813033    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:29:58.348547 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.348517    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"]
Jul 24 21:29:58.352332 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.352307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.354845 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.354827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 24 21:29:58.354931 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.354832    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 24 21:29:58.364694 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.364670    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"]
Jul 24 21:29:58.485742 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.485709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwgw\" (UniqueName: \"kubernetes.io/projected/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-api-access-xvwgw\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.485915 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.485750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.485915 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.485795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a73378bc-5e96-4f03-884f-b6c8813479ac-kserve-provision-location\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.485915 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.485843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-cabundle-cert\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.485915 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.485873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a73378bc-5e96-4f03-884f-b6c8813479ac-proxy-tls\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.586356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.586318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwgw\" (UniqueName: \"kubernetes.io/projected/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-api-access-xvwgw\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.586576 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.586368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.586576 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.586415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a73378bc-5e96-4f03-884f-b6c8813479ac-kserve-provision-location\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.586576 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.586453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-cabundle-cert\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.586576 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.586482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a73378bc-5e96-4f03-884f-b6c8813479ac-proxy-tls\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.586949 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.586927    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a73378bc-5e96-4f03-884f-b6c8813479ac-kserve-provision-location\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.587127 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.587109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-cabundle-cert\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.587175 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.587139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.589102 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.589083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a73378bc-5e96-4f03-884f-b6c8813479ac-proxy-tls\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.595663 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.595630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwgw\" (UniqueName: \"kubernetes.io/projected/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-api-access-xvwgw\") pod \"isvc-logger-predictor-5f586b99d8-ckct4\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.663214 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.663141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:29:58.800982 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:58.800959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"]
Jul 24 21:29:58.803059 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:29:58.803028    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73378bc_5e96_4f03_884f_b6c8813479ac.slice/crio-2100446467a31aa54a3afd796f07816e4de93c0673ad33a6ea485e9ac5873482 WatchSource:0}: Error finding container 2100446467a31aa54a3afd796f07816e4de93c0673ad33a6ea485e9ac5873482: Status 404 returned error can't find the container with id 2100446467a31aa54a3afd796f07816e4de93c0673ad33a6ea485e9ac5873482
Jul 24 21:29:59.340133 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:59.340088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerStarted","Data":"d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a"}
Jul 24 21:29:59.340301 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:29:59.340216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerStarted","Data":"2100446467a31aa54a3afd796f07816e4de93c0673ad33a6ea485e9ac5873482"}
Jul 24 21:30:01.806973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:01.806934    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 24 21:30:03.360555 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:03.360525    2580 generic.go:356] "Generic (PLEG): container finished" podID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerID="d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a" exitCode=0
Jul 24 21:30:03.360940 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:03.360602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerDied","Data":"d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a"}
Jul 24 21:30:04.368361 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:04.368326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerStarted","Data":"e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5"}
Jul 24 21:30:04.368765 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:04.368419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerStarted","Data":"41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4"}
Jul 24 21:30:04.368765 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:04.368446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerStarted","Data":"4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71"}
Jul 24 21:30:04.368765 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:04.368726    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:30:04.368891 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:04.368857    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:30:04.370007 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:04.369976    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:30:04.392917 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:04.392861    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podStartSLOduration=2.8096308370000003 podStartE2EDuration="6.392844073s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.583213236s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:30:04.39009296 +0000 UTC m=+878.728934760" watchObservedRunningTime="2026-07-24 21:30:04.392844073 +0000 UTC m=+878.731685848"
Jul 24 21:30:05.374475 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:05.374427    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:30:05.374940 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:05.374452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:30:05.375622 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:05.375595    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:30:06.379715 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:06.379673    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:30:06.380284 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:06.380256    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:30:06.806849 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:06.806808    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 24 21:30:06.811528 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:06.811498    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:30:06.811653 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:06.811637    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:30:06.812703 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:06.812682    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 24 21:30:06.812794 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:06.812773    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:30:08.392099 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.392065    2580 generic.go:356] "Generic (PLEG): container finished" podID="f30dae70-b697-41b6-bda3-0319d839261a" containerID="b34c985e853a4b8d461899c20f6ddbaac0dc88e28db4f13ba2fdc7c84d93aa9f" exitCode=0
Jul 24 21:30:08.392484 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.392148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerDied","Data":"b34c985e853a4b8d461899c20f6ddbaac0dc88e28db4f13ba2fdc7c84d93aa9f"}
Jul 24 21:30:08.519544 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.519517    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:30:08.682601 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.682519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-kube-rbac-proxy-sar-config\") pod \"f30dae70-b697-41b6-bda3-0319d839261a\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") "
Jul 24 21:30:08.682601 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.682582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30dae70-b697-41b6-bda3-0319d839261a-kserve-provision-location\") pod \"f30dae70-b697-41b6-bda3-0319d839261a\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") "
Jul 24 21:30:08.682795 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.682637    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-cabundle-cert\") pod \"f30dae70-b697-41b6-bda3-0319d839261a\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") "
Jul 24 21:30:08.682795 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.682653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lrqz\" (UniqueName: \"kubernetes.io/projected/f30dae70-b697-41b6-bda3-0319d839261a-kube-api-access-7lrqz\") pod \"f30dae70-b697-41b6-bda3-0319d839261a\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") "
Jul 24 21:30:08.682795 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.682681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f30dae70-b697-41b6-bda3-0319d839261a-proxy-tls\") pod \"f30dae70-b697-41b6-bda3-0319d839261a\" (UID: \"f30dae70-b697-41b6-bda3-0319d839261a\") "
Jul 24 21:30:08.683024 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.683002    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f30dae70-b697-41b6-bda3-0319d839261a" (UID: "f30dae70-b697-41b6-bda3-0319d839261a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:30:08.683024 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.683006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30dae70-b697-41b6-bda3-0319d839261a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f30dae70-b697-41b6-bda3-0319d839261a" (UID: "f30dae70-b697-41b6-bda3-0319d839261a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:30:08.683140 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.683037    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f30dae70-b697-41b6-bda3-0319d839261a" (UID: "f30dae70-b697-41b6-bda3-0319d839261a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:30:08.684970 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.684950    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30dae70-b697-41b6-bda3-0319d839261a-kube-api-access-7lrqz" (OuterVolumeSpecName: "kube-api-access-7lrqz") pod "f30dae70-b697-41b6-bda3-0319d839261a" (UID: "f30dae70-b697-41b6-bda3-0319d839261a"). InnerVolumeSpecName "kube-api-access-7lrqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:30:08.685035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.684971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30dae70-b697-41b6-bda3-0319d839261a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f30dae70-b697-41b6-bda3-0319d839261a" (UID: "f30dae70-b697-41b6-bda3-0319d839261a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:30:08.783263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.783217    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f30dae70-b697-41b6-bda3-0319d839261a-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:30:08.783263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.783258    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:30:08.783263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.783269    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30dae70-b697-41b6-bda3-0319d839261a-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:30:08.783263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.783279    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30dae70-b697-41b6-bda3-0319d839261a-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:30:08.783554 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:08.783287    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrqz\" (UniqueName: \"kubernetes.io/projected/f30dae70-b697-41b6-bda3-0319d839261a-kube-api-access-7lrqz\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:30:09.399125 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:09.399090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97" event={"ID":"f30dae70-b697-41b6-bda3-0319d839261a","Type":"ContainerDied","Data":"bf1e7ec6ba85ed0fe2327fe4f4febf52fe9a31b50e70925616eeeb60fdc8d57c"}
Jul 24 21:30:09.399569 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:09.399162    2580 scope.go:117] "RemoveContainer" containerID="b34c985e853a4b8d461899c20f6ddbaac0dc88e28db4f13ba2fdc7c84d93aa9f"
Jul 24 21:30:09.399569 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:09.399101    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"
Jul 24 21:30:09.408153 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:09.408136    2580 scope.go:117] "RemoveContainer" containerID="1ec2adabaef1d7687e6680448fd9f488b34d6cd51446400f4ad921ce0b266fbc"
Jul 24 21:30:09.416414 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:09.416375    2580 scope.go:117] "RemoveContainer" containerID="f66d8060a00b6f415ddb98569119bef77f100fa8f3508d39b4198db28ad07498"
Jul 24 21:30:09.423301 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:09.423272    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"]
Jul 24 21:30:09.425054 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:09.425000    2580 scope.go:117] "RemoveContainer" containerID="79c75406ee8f4a2d4de668671d52574ea43cc6faff55250f927c7247c91cff3c"
Jul 24 21:30:09.428033 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:09.428012    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-694d6c969d-qqj97"]
Jul 24 21:30:10.287582 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:10.287548    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f30dae70-b697-41b6-bda3-0319d839261a" path="/var/lib/kubelet/pods/f30dae70-b697-41b6-bda3-0319d839261a/volumes"
Jul 24 21:30:11.383930 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:11.383894    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:30:11.384611 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:11.384580    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:30:11.384835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:11.384791    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:30:21.384661 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:21.384575    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:30:21.385080 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:21.385058    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:30:26.141182 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:26.141153    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:30:26.145870 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:26.145848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:30:31.384586 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:31.384543    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:30:31.385060 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:31.384950    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:30:41.385040 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:41.385003    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:30:41.385551 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:41.385422    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:30:51.384514 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:51.384465    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:30:51.384973 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:30:51.384871    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:31:01.384845 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:01.384804    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:31:01.385311 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:01.385289    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:31:11.384528 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:11.384488    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:31:11.384999 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:11.384960    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:31:16.293649 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:16.293620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:31:16.294128 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:16.293708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:31:23.386319 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.386287    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-pfc7l_27039753-bd28-4b3f-bed8-18259c111ae2/kserve-container/0.log"
Jul 24 21:31:23.586543 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.586504    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"]
Jul 24 21:31:23.586906 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.586851    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" containerID="cri-o://4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71" gracePeriod=30
Jul 24 21:31:23.587164 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.586880    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" containerID="cri-o://e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5" gracePeriod=30
Jul 24 21:31:23.587274 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.586915    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy" containerID="cri-o://41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4" gracePeriod=30
Jul 24 21:31:23.658883 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.658803    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"]
Jul 24 21:31:23.659043 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.659013    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l" podUID="27039753-bd28-4b3f-bed8-18259c111ae2" containerName="kserve-container" containerID="cri-o://021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc" gracePeriod=30
Jul 24 21:31:23.659163 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.659068    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l" podUID="27039753-bd28-4b3f-bed8-18259c111ae2" containerName="kube-rbac-proxy" containerID="cri-o://c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485" gracePeriod=30
Jul 24 21:31:23.675423 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675380    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"]
Jul 24 21:31:23.675833 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675818    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="storage-initializer"
Jul 24 21:31:23.675886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675836    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="storage-initializer"
Jul 24 21:31:23.675886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675850    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy"
Jul 24 21:31:23.675886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675856    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy"
Jul 24 21:31:23.675886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675872    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container"
Jul 24 21:31:23.675886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675878    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container"
Jul 24 21:31:23.676160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675888    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent"
Jul 24 21:31:23.676160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675896    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent"
Jul 24 21:31:23.676160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675974    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="agent"
Jul 24 21:31:23.676160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675981    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kube-rbac-proxy"
Jul 24 21:31:23.676160 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.675988    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30dae70-b697-41b6-bda3-0319d839261a" containerName="kserve-container"
Jul 24 21:31:23.679162 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.679144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.682518 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.682499    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 24 21:31:23.683476 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.683461    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 24 21:31:23.691101 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.691075    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"]
Jul 24 21:31:23.739184 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.739156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/577376b7-4372-4a36-a084-b2e594aacf5b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.739351 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.739206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgl2\" (UniqueName: \"kubernetes.io/projected/577376b7-4372-4a36-a084-b2e594aacf5b-kube-api-access-mfgl2\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.739351 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.739238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/577376b7-4372-4a36-a084-b2e594aacf5b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.739351 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.739256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.739351 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.739273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.761497 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.761467    2580 generic.go:356] "Generic (PLEG): container finished" podID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerID="41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4" exitCode=2
Jul 24 21:31:23.761667 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.761532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerDied","Data":"41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4"}
Jul 24 21:31:23.840067 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.840014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/577376b7-4372-4a36-a084-b2e594aacf5b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.840223 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.840114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgl2\" (UniqueName: \"kubernetes.io/projected/577376b7-4372-4a36-a084-b2e594aacf5b-kube-api-access-mfgl2\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.840223 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.840177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/577376b7-4372-4a36-a084-b2e594aacf5b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.840223 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.840207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.840434 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.840243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.840533 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.840501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/577376b7-4372-4a36-a084-b2e594aacf5b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.840641 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:31:23.840380    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 24 21:31:23.840708 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:31:23.840699    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577376b7-4372-4a36-a084-b2e594aacf5b-proxy-tls podName:577376b7-4372-4a36-a084-b2e594aacf5b nodeName:}" failed. No retries permitted until 2026-07-24 21:31:24.340677358 +0000 UTC m=+958.679519114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/577376b7-4372-4a36-a084-b2e594aacf5b-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-76zsh" (UID: "577376b7-4372-4a36-a084-b2e594aacf5b") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 24 21:31:23.840991 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.840962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.840991 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.840983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.849906 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.849877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgl2\" (UniqueName: \"kubernetes.io/projected/577376b7-4372-4a36-a084-b2e594aacf5b-kube-api-access-mfgl2\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:23.901685 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.901657    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:31:23.941269 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.941200    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27039753-bd28-4b3f-bed8-18259c111ae2-proxy-tls\") pod \"27039753-bd28-4b3f-bed8-18259c111ae2\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") "
Jul 24 21:31:23.941269 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.941242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8zk6\" (UniqueName: \"kubernetes.io/projected/27039753-bd28-4b3f-bed8-18259c111ae2-kube-api-access-k8zk6\") pod \"27039753-bd28-4b3f-bed8-18259c111ae2\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") "
Jul 24 21:31:23.941466 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.941278    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27039753-bd28-4b3f-bed8-18259c111ae2-kube-rbac-proxy-sar-config\") pod \"27039753-bd28-4b3f-bed8-18259c111ae2\" (UID: \"27039753-bd28-4b3f-bed8-18259c111ae2\") "
Jul 24 21:31:23.941701 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.941679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27039753-bd28-4b3f-bed8-18259c111ae2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27039753-bd28-4b3f-bed8-18259c111ae2" (UID: "27039753-bd28-4b3f-bed8-18259c111ae2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:31:23.943528 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.943503    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27039753-bd28-4b3f-bed8-18259c111ae2-kube-api-access-k8zk6" (OuterVolumeSpecName: "kube-api-access-k8zk6") pod "27039753-bd28-4b3f-bed8-18259c111ae2" (UID: "27039753-bd28-4b3f-bed8-18259c111ae2"). InnerVolumeSpecName "kube-api-access-k8zk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:31:23.943617 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:23.943505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27039753-bd28-4b3f-bed8-18259c111ae2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27039753-bd28-4b3f-bed8-18259c111ae2" (UID: "27039753-bd28-4b3f-bed8-18259c111ae2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:31:24.042308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.042278    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27039753-bd28-4b3f-bed8-18259c111ae2-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:31:24.042308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.042302    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8zk6\" (UniqueName: \"kubernetes.io/projected/27039753-bd28-4b3f-bed8-18259c111ae2-kube-api-access-k8zk6\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:31:24.042308 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.042312    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27039753-bd28-4b3f-bed8-18259c111ae2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:31:24.345862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.345830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/577376b7-4372-4a36-a084-b2e594aacf5b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:24.348366 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.348337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/577376b7-4372-4a36-a084-b2e594aacf5b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-76zsh\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:24.631478 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.631446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:31:24.767952 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.767913    2580 generic.go:356] "Generic (PLEG): container finished" podID="27039753-bd28-4b3f-bed8-18259c111ae2" containerID="c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485" exitCode=2
Jul 24 21:31:24.767952 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.767945    2580 generic.go:356] "Generic (PLEG): container finished" podID="27039753-bd28-4b3f-bed8-18259c111ae2" containerID="021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc" exitCode=2
Jul 24 21:31:24.768227 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.767972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l" event={"ID":"27039753-bd28-4b3f-bed8-18259c111ae2","Type":"ContainerDied","Data":"c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485"}
Jul 24 21:31:24.768227 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.767991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"
Jul 24 21:31:24.768227 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.768030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l" event={"ID":"27039753-bd28-4b3f-bed8-18259c111ae2","Type":"ContainerDied","Data":"021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc"}
Jul 24 21:31:24.768227 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.768063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l" event={"ID":"27039753-bd28-4b3f-bed8-18259c111ae2","Type":"ContainerDied","Data":"e4948b05cb22bed157865830f0dd111bca0d071af0f96b2ac8f218e64b8e3d8c"}
Jul 24 21:31:24.768227 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.768101    2580 scope.go:117] "RemoveContainer" containerID="c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485"
Jul 24 21:31:24.770323 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.770302    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"]
Jul 24 21:31:24.773671 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:31:24.773650    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577376b7_4372_4a36_a084_b2e594aacf5b.slice/crio-016f6f0481ec1931349bcd2d01991076b4618a9e3802ed7d6b4da0afa343d858 WatchSource:0}: Error finding container 016f6f0481ec1931349bcd2d01991076b4618a9e3802ed7d6b4da0afa343d858: Status 404 returned error can't find the container with id 016f6f0481ec1931349bcd2d01991076b4618a9e3802ed7d6b4da0afa343d858
Jul 24 21:31:24.783866 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.783838    2580 scope.go:117] "RemoveContainer" containerID="021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc"
Jul 24 21:31:24.791520 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.791498    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"]
Jul 24 21:31:24.792976 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.792956    2580 scope.go:117] "RemoveContainer" containerID="c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485"
Jul 24 21:31:24.793247 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:31:24.793228    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485\": container with ID starting with c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485 not found: ID does not exist" containerID="c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485"
Jul 24 21:31:24.793338 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.793261    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485"} err="failed to get container status \"c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485\": rpc error: code = NotFound desc = could not find container \"c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485\": container with ID starting with c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485 not found: ID does not exist"
Jul 24 21:31:24.793338 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.793280    2580 scope.go:117] "RemoveContainer" containerID="021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc"
Jul 24 21:31:24.793589 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:31:24.793566    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc\": container with ID starting with 021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc not found: ID does not exist" containerID="021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc"
Jul 24 21:31:24.793636 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.793597    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc"} err="failed to get container status \"021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc\": rpc error: code = NotFound desc = could not find container \"021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc\": container with ID starting with 021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc not found: ID does not exist"
Jul 24 21:31:24.793636 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.793617    2580 scope.go:117] "RemoveContainer" containerID="c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485"
Jul 24 21:31:24.793898 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.793878    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485"} err="failed to get container status \"c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485\": rpc error: code = NotFound desc = could not find container \"c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485\": container with ID starting with c07bb0dd7dbaf8d2bc93aa25166c4aeb62eccccebe36955d85f20fe386fa7485 not found: ID does not exist"
Jul 24 21:31:24.793898 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.793901    2580 scope.go:117] "RemoveContainer" containerID="021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc"
Jul 24 21:31:24.794177 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.794143    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc"} err="failed to get container status \"021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc\": rpc error: code = NotFound desc = could not find container \"021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc\": container with ID starting with 021ec08b0ecf7fc31050177bf57e80d6d7c55129748f40a8ce202ee3072885fc not found: ID does not exist"
Jul 24 21:31:24.796642 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:24.796622    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-pfc7l"]
Jul 24 21:31:25.774078 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:25.774037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" event={"ID":"577376b7-4372-4a36-a084-b2e594aacf5b","Type":"ContainerStarted","Data":"3629de3da9486505af258c852164672c071b6bc72c4945ac17fc9fc41bee5f8f"}
Jul 24 21:31:25.774527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:25.774099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" event={"ID":"577376b7-4372-4a36-a084-b2e594aacf5b","Type":"ContainerStarted","Data":"016f6f0481ec1931349bcd2d01991076b4618a9e3802ed7d6b4da0afa343d858"}
Jul 24 21:31:26.284694 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:26.284652    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:31:26.285060 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:26.285036    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:31:26.287350 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:26.287327    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27039753-bd28-4b3f-bed8-18259c111ae2" path="/var/lib/kubelet/pods/27039753-bd28-4b3f-bed8-18259c111ae2/volumes"
Jul 24 21:31:26.380075 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:26.380030    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 24 21:31:28.790984 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:28.790954    2580 generic.go:356] "Generic (PLEG): container finished" podID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerID="4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71" exitCode=0
Jul 24 21:31:28.791339 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:28.791031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerDied","Data":"4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71"}
Jul 24 21:31:31.380367 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:31.380328    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 24 21:31:36.284831 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:36.284776    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:31:36.285299 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:36.285093    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:31:36.380276 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:36.380228    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 24 21:31:36.380452 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:36.380360    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:31:41.380656 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:41.380614    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 24 21:31:45.873757 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:45.873721    2580 generic.go:356] "Generic (PLEG): container finished" podID="577376b7-4372-4a36-a084-b2e594aacf5b" containerID="3629de3da9486505af258c852164672c071b6bc72c4945ac17fc9fc41bee5f8f" exitCode=0
Jul 24 21:31:45.874144 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:45.873772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" event={"ID":"577376b7-4372-4a36-a084-b2e594aacf5b","Type":"ContainerDied","Data":"3629de3da9486505af258c852164672c071b6bc72c4945ac17fc9fc41bee5f8f"}
Jul 24 21:31:46.284718 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:46.284680    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:31:46.285052 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:46.285024    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:31:46.287607 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:46.287586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:31:46.287729 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:46.287677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:31:46.380145 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:46.380103    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 24 21:31:51.380612 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:51.380562    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 24 21:31:53.762718 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.762679    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:31:53.921490 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.921401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvwgw\" (UniqueName: \"kubernetes.io/projected/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-api-access-xvwgw\") pod \"a73378bc-5e96-4f03-884f-b6c8813479ac\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") "
Jul 24 21:31:53.921490 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.921439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a73378bc-5e96-4f03-884f-b6c8813479ac-proxy-tls\") pod \"a73378bc-5e96-4f03-884f-b6c8813479ac\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") "
Jul 24 21:31:53.921713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.921530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-rbac-proxy-sar-config\") pod \"a73378bc-5e96-4f03-884f-b6c8813479ac\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") "
Jul 24 21:31:53.921713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.921578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a73378bc-5e96-4f03-884f-b6c8813479ac-kserve-provision-location\") pod \"a73378bc-5e96-4f03-884f-b6c8813479ac\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") "
Jul 24 21:31:53.921713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.921625    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-cabundle-cert\") pod \"a73378bc-5e96-4f03-884f-b6c8813479ac\" (UID: \"a73378bc-5e96-4f03-884f-b6c8813479ac\") "
Jul 24 21:31:53.921998 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.921971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a73378bc-5e96-4f03-884f-b6c8813479ac" (UID: "a73378bc-5e96-4f03-884f-b6c8813479ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:31:53.922083 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.921971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73378bc-5e96-4f03-884f-b6c8813479ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a73378bc-5e96-4f03-884f-b6c8813479ac" (UID: "a73378bc-5e96-4f03-884f-b6c8813479ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:31:53.922148 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.922116    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a73378bc-5e96-4f03-884f-b6c8813479ac" (UID: "a73378bc-5e96-4f03-884f-b6c8813479ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:31:53.924067 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.924025    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-api-access-xvwgw" (OuterVolumeSpecName: "kube-api-access-xvwgw") pod "a73378bc-5e96-4f03-884f-b6c8813479ac" (UID: "a73378bc-5e96-4f03-884f-b6c8813479ac"). InnerVolumeSpecName "kube-api-access-xvwgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:31:53.924198 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.924131    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73378bc-5e96-4f03-884f-b6c8813479ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a73378bc-5e96-4f03-884f-b6c8813479ac" (UID: "a73378bc-5e96-4f03-884f-b6c8813479ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:31:53.932423 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.932375    2580 generic.go:356] "Generic (PLEG): container finished" podID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerID="e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5" exitCode=0
Jul 24 21:31:53.932532 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.932455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerDied","Data":"e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5"}
Jul 24 21:31:53.932532 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.932476    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"
Jul 24 21:31:53.932635 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.932538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4" event={"ID":"a73378bc-5e96-4f03-884f-b6c8813479ac","Type":"ContainerDied","Data":"2100446467a31aa54a3afd796f07816e4de93c0673ad33a6ea485e9ac5873482"}
Jul 24 21:31:53.932635 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.932628    2580 scope.go:117] "RemoveContainer" containerID="e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5"
Jul 24 21:31:53.944577 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.944561    2580 scope.go:117] "RemoveContainer" containerID="41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4"
Jul 24 21:31:53.955004 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.954861    2580 scope.go:117] "RemoveContainer" containerID="4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71"
Jul 24 21:31:53.958169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.958145    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"]
Jul 24 21:31:53.961783 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.961762    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5f586b99d8-ckct4"]
Jul 24 21:31:53.964949 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.964931    2580 scope.go:117] "RemoveContainer" containerID="d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a"
Jul 24 21:31:53.973274 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.973252    2580 scope.go:117] "RemoveContainer" containerID="e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5"
Jul 24 21:31:53.973618 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:31:53.973597    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5\": container with ID starting with e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5 not found: ID does not exist" containerID="e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5"
Jul 24 21:31:53.973716 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.973630    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5"} err="failed to get container status \"e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5\": rpc error: code = NotFound desc = could not find container \"e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5\": container with ID starting with e98025fce8f984a333cabd6afbefcde887c760254cee9fc51c58e8014abd3dc5 not found: ID does not exist"
Jul 24 21:31:53.973716 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.973657    2580 scope.go:117] "RemoveContainer" containerID="41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4"
Jul 24 21:31:53.973949 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:31:53.973930    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4\": container with ID starting with 41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4 not found: ID does not exist" containerID="41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4"
Jul 24 21:31:53.973997 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.973957    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4"} err="failed to get container status \"41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4\": rpc error: code = NotFound desc = could not find container \"41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4\": container with ID starting with 41534d85a020f4e397968f951c76d9143c18b08826c38dfb7e22a7f3cb9bd9e4 not found: ID does not exist"
Jul 24 21:31:53.973997 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.973981    2580 scope.go:117] "RemoveContainer" containerID="4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71"
Jul 24 21:31:53.974258 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:31:53.974237    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71\": container with ID starting with 4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71 not found: ID does not exist" containerID="4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71"
Jul 24 21:31:53.974359 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.974262    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71"} err="failed to get container status \"4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71\": rpc error: code = NotFound desc = could not find container \"4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71\": container with ID starting with 4db1e664762243f1072481fd08eeaa50570227f82bd3455414ab3aa3a7d64b71 not found: ID does not exist"
Jul 24 21:31:53.974359 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.974277    2580 scope.go:117] "RemoveContainer" containerID="d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a"
Jul 24 21:31:53.974618 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:31:53.974598    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a\": container with ID starting with d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a not found: ID does not exist" containerID="d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a"
Jul 24 21:31:53.974679 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:53.974632    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a"} err="failed to get container status \"d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a\": rpc error: code = NotFound desc = could not find container \"d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a\": container with ID starting with d9710ba8326cd99c52fefb1c2b95f2a0d589e5438732a1f4d85a8e9605606f2a not found: ID does not exist"
Jul 24 21:31:54.022889 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:54.022848    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:31:54.022889 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:54.022887    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a73378bc-5e96-4f03-884f-b6c8813479ac-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:31:54.022889 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:54.022897    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a73378bc-5e96-4f03-884f-b6c8813479ac-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:31:54.023129 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:54.022907    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwgw\" (UniqueName: \"kubernetes.io/projected/a73378bc-5e96-4f03-884f-b6c8813479ac-kube-api-access-xvwgw\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:31:54.023129 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:54.022918    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a73378bc-5e96-4f03-884f-b6c8813479ac-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:31:54.288292 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:31:54.288260    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" path="/var/lib/kubelet/pods/a73378bc-5e96-4f03-884f-b6c8813479ac/volumes"
Jul 24 21:33:48.513294 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:33:48.513255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" event={"ID":"577376b7-4372-4a36-a084-b2e594aacf5b","Type":"ContainerStarted","Data":"fc544a12f851084011a0f8b909b2019036f86b54a5bd68c8797d8024795a7cf8"}
Jul 24 21:33:48.513740 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:33:48.513325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" event={"ID":"577376b7-4372-4a36-a084-b2e594aacf5b","Type":"ContainerStarted","Data":"a9e6e8c9c618b453d722a002dcb11a5dc8e4abb1b9279cf63e186f880d0a5a20"}
Jul 24 21:33:48.513740 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:33:48.513423    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:33:48.549710 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:33:48.549647    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" podStartSLOduration=3.116131592 podStartE2EDuration="2m25.549620801s" totalImagesPullingTime="2m1.757422504s" totalInitContainerRuntime="20.676066705s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:33:48.545542201 +0000 UTC m=+1102.884383976" watchObservedRunningTime="2026-07-24 21:33:48.549620801 +0000 UTC m=+1102.888462585"
Jul 24 21:33:49.525270 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:33:49.525230    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:33:49.526791 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:33:49.526747    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:33:50.528905 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:33:50.528865    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:33:55.533985 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:33:55.533958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:33:55.534605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:33:55.534492    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:34:05.535346 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:05.535317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:34:20.377596 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:20.377564    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"]
Jul 24 21:34:20.378048 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:20.377826    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kserve-container" containerID="cri-o://a9e6e8c9c618b453d722a002dcb11a5dc8e4abb1b9279cf63e186f880d0a5a20" gracePeriod=30
Jul 24 21:34:20.378048 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:20.377885    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kube-rbac-proxy" containerID="cri-o://fc544a12f851084011a0f8b909b2019036f86b54a5bd68c8797d8024795a7cf8" gracePeriod=30
Jul 24 21:34:20.530149 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:20.530109    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 24 21:34:20.661877 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:20.661788    2580 generic.go:356] "Generic (PLEG): container finished" podID="577376b7-4372-4a36-a084-b2e594aacf5b" containerID="fc544a12f851084011a0f8b909b2019036f86b54a5bd68c8797d8024795a7cf8" exitCode=2
Jul 24 21:34:20.662014 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:20.661867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" event={"ID":"577376b7-4372-4a36-a084-b2e594aacf5b","Type":"ContainerDied","Data":"fc544a12f851084011a0f8b909b2019036f86b54a5bd68c8797d8024795a7cf8"}
Jul 24 21:34:21.669622 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.669592    2580 generic.go:356] "Generic (PLEG): container finished" podID="577376b7-4372-4a36-a084-b2e594aacf5b" containerID="a9e6e8c9c618b453d722a002dcb11a5dc8e4abb1b9279cf63e186f880d0a5a20" exitCode=0
Jul 24 21:34:21.670064 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.669650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" event={"ID":"577376b7-4372-4a36-a084-b2e594aacf5b","Type":"ContainerDied","Data":"a9e6e8c9c618b453d722a002dcb11a5dc8e4abb1b9279cf63e186f880d0a5a20"}
Jul 24 21:34:21.758805 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.758769    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:34:21.810221 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.810195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/577376b7-4372-4a36-a084-b2e594aacf5b-proxy-tls\") pod \"577376b7-4372-4a36-a084-b2e594aacf5b\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") "
Jul 24 21:34:21.810419 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.810230    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/577376b7-4372-4a36-a084-b2e594aacf5b-kserve-provision-location\") pod \"577376b7-4372-4a36-a084-b2e594aacf5b\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") "
Jul 24 21:34:21.810419 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.810257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-cabundle-cert\") pod \"577376b7-4372-4a36-a084-b2e594aacf5b\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") "
Jul 24 21:34:21.810419 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.810286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-kube-rbac-proxy-sar-config\") pod \"577376b7-4372-4a36-a084-b2e594aacf5b\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") "
Jul 24 21:34:21.810419 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.810332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfgl2\" (UniqueName: \"kubernetes.io/projected/577376b7-4372-4a36-a084-b2e594aacf5b-kube-api-access-mfgl2\") pod \"577376b7-4372-4a36-a084-b2e594aacf5b\" (UID: \"577376b7-4372-4a36-a084-b2e594aacf5b\") "
Jul 24 21:34:21.810680 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.810641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "577376b7-4372-4a36-a084-b2e594aacf5b" (UID: "577376b7-4372-4a36-a084-b2e594aacf5b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:34:21.810764 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.810736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "577376b7-4372-4a36-a084-b2e594aacf5b" (UID: "577376b7-4372-4a36-a084-b2e594aacf5b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:34:21.812461 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.812440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577376b7-4372-4a36-a084-b2e594aacf5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "577376b7-4372-4a36-a084-b2e594aacf5b" (UID: "577376b7-4372-4a36-a084-b2e594aacf5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:34:21.812755 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.812738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577376b7-4372-4a36-a084-b2e594aacf5b-kube-api-access-mfgl2" (OuterVolumeSpecName: "kube-api-access-mfgl2") pod "577376b7-4372-4a36-a084-b2e594aacf5b" (UID: "577376b7-4372-4a36-a084-b2e594aacf5b"). InnerVolumeSpecName "kube-api-access-mfgl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:34:21.815983 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.815957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577376b7-4372-4a36-a084-b2e594aacf5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "577376b7-4372-4a36-a084-b2e594aacf5b" (UID: "577376b7-4372-4a36-a084-b2e594aacf5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:34:21.911835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.911747    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/577376b7-4372-4a36-a084-b2e594aacf5b-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:34:21.911835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.911775    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/577376b7-4372-4a36-a084-b2e594aacf5b-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:34:21.911835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.911787    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:34:21.911835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.911798    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/577376b7-4372-4a36-a084-b2e594aacf5b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:34:21.911835 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:21.911809    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfgl2\" (UniqueName: \"kubernetes.io/projected/577376b7-4372-4a36-a084-b2e594aacf5b-kube-api-access-mfgl2\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:34:22.675856 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:22.675770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh" event={"ID":"577376b7-4372-4a36-a084-b2e594aacf5b","Type":"ContainerDied","Data":"016f6f0481ec1931349bcd2d01991076b4618a9e3802ed7d6b4da0afa343d858"}
Jul 24 21:34:22.675856 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:22.675846    2580 scope.go:117] "RemoveContainer" containerID="fc544a12f851084011a0f8b909b2019036f86b54a5bd68c8797d8024795a7cf8"
Jul 24 21:34:22.676345 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:22.675787    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"
Jul 24 21:34:22.684719 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:22.684699    2580 scope.go:117] "RemoveContainer" containerID="a9e6e8c9c618b453d722a002dcb11a5dc8e4abb1b9279cf63e186f880d0a5a20"
Jul 24 21:34:22.692344 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:22.692328    2580 scope.go:117] "RemoveContainer" containerID="3629de3da9486505af258c852164672c071b6bc72c4945ac17fc9fc41bee5f8f"
Jul 24 21:34:22.697528 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:22.697507    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"]
Jul 24 21:34:22.701032 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:22.701012    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-76zsh"]
Jul 24 21:34:24.287061 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:24.287025    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" path="/var/lib/kubelet/pods/577376b7-4372-4a36-a084-b2e594aacf5b/volumes"
Jul 24 21:34:25.531783 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.531752    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"]
Jul 24 21:34:25.532151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532086    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27039753-bd28-4b3f-bed8-18259c111ae2" containerName="kube-rbac-proxy"
Jul 24 21:34:25.532151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532098    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27039753-bd28-4b3f-bed8-18259c111ae2" containerName="kube-rbac-proxy"
Jul 24 21:34:25.532151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532111    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy"
Jul 24 21:34:25.532151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532116    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy"
Jul 24 21:34:25.532151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532125    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="storage-initializer"
Jul 24 21:34:25.532151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532130    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="storage-initializer"
Jul 24 21:34:25.532151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532137    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="storage-initializer"
Jul 24 21:34:25.532151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532142    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="storage-initializer"
Jul 24 21:34:25.532151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532152    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27039753-bd28-4b3f-bed8-18259c111ae2" containerName="kserve-container"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532157    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27039753-bd28-4b3f-bed8-18259c111ae2" containerName="kserve-container"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532162    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532169    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532178    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kserve-container"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532183    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kserve-container"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532192    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532197    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532203    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kube-rbac-proxy"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532207    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kube-rbac-proxy"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532256    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kube-rbac-proxy"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532262    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="kserve-container"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532271    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27039753-bd28-4b3f-bed8-18259c111ae2" containerName="kserve-container"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532276    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kserve-container"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532282    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="577376b7-4372-4a36-a084-b2e594aacf5b" containerName="kube-rbac-proxy"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532289    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27039753-bd28-4b3f-bed8-18259c111ae2" containerName="kube-rbac-proxy"
Jul 24 21:34:25.532447 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.532295    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73378bc-5e96-4f03-884f-b6c8813479ac" containerName="agent"
Jul 24 21:34:25.535588 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.535573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.538073 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.538052    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:34:25.538210 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.538105    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 24 21:34:25.538210 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.538162    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 24 21:34:25.538210 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.538185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:34:25.538366 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.538348    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:34:25.538434 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.538366    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:34:25.539087 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.539073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:34:25.546053 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.546030    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"]
Jul 24 21:34:25.642780 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.642730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.642780 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.642795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhlm7\" (UniqueName: \"kubernetes.io/projected/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-api-access-dhlm7\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.643069 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.642852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.643069 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.642949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.643069 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.642975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.743767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.743728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.743767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.743767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhlm7\" (UniqueName: \"kubernetes.io/projected/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-api-access-dhlm7\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.744044 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.743788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.744044 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.743828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.744044 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.743844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.744254 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.744230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.744653 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.744632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.744700 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.744632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.746498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.746476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.753462 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.753435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhlm7\" (UniqueName: \"kubernetes.io/projected/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-api-access-dhlm7\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9g7nc\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.846721 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.846635    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:25.983306 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:25.983269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"]
Jul 24 21:34:25.987549 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:34:25.987523    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328ae144_7d03_4fd8_93d4_fc2ffaf71c16.slice/crio-e79be5f71e2cdc053d5a5c42233043ca1bc73c6f3eb4b48391e392be3f66cc92 WatchSource:0}: Error finding container e79be5f71e2cdc053d5a5c42233043ca1bc73c6f3eb4b48391e392be3f66cc92: Status 404 returned error can't find the container with id e79be5f71e2cdc053d5a5c42233043ca1bc73c6f3eb4b48391e392be3f66cc92
Jul 24 21:34:26.694307 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:26.694267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" event={"ID":"328ae144-7d03-4fd8-93d4-fc2ffaf71c16","Type":"ContainerStarted","Data":"f1f30d431198e0c5fac025bc50ac656f1e45a2a9c8fd39a44d34efc3019aad36"}
Jul 24 21:34:26.694691 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:26.694335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" event={"ID":"328ae144-7d03-4fd8-93d4-fc2ffaf71c16","Type":"ContainerStarted","Data":"e79be5f71e2cdc053d5a5c42233043ca1bc73c6f3eb4b48391e392be3f66cc92"}
Jul 24 21:34:46.788529 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:46.788496    2580 generic.go:356] "Generic (PLEG): container finished" podID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerID="f1f30d431198e0c5fac025bc50ac656f1e45a2a9c8fd39a44d34efc3019aad36" exitCode=0
Jul 24 21:34:46.788953 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:46.788545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" event={"ID":"328ae144-7d03-4fd8-93d4-fc2ffaf71c16","Type":"ContainerDied","Data":"f1f30d431198e0c5fac025bc50ac656f1e45a2a9c8fd39a44d34efc3019aad36"}
Jul 24 21:34:47.795864 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:47.795832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" event={"ID":"328ae144-7d03-4fd8-93d4-fc2ffaf71c16","Type":"ContainerStarted","Data":"44c017c134cf3b6916afb659ad97ae4ac54e99250f500d993fc7ce39007f5f55"}
Jul 24 21:34:47.796345 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:47.795892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" event={"ID":"328ae144-7d03-4fd8-93d4-fc2ffaf71c16","Type":"ContainerStarted","Data":"7d97acc9fa1328226355b49439994017c0ebc146c80128a92be8b9e5599cdaba"}
Jul 24 21:34:47.796345 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:47.796069    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:47.821878 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:47.821819    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" podStartSLOduration=2.261211849 podStartE2EDuration="22.82180288s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.560591031s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:34:47.818939537 +0000 UTC m=+1162.157781314" watchObservedRunningTime="2026-07-24 21:34:47.82180288 +0000 UTC m=+1162.160644656"
Jul 24 21:34:48.800977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:48.800942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:34:54.812538 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:34:54.812502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:35:26.174146 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:35:26.174117    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:35:26.176045 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:35:26.176015    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:36:16.290464 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:16.290363    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:36:27.329584 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:27.329523    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.34:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 21:36:31.971402 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:31.971351    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"]
Jul 24 21:36:31.971872 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:31.971682    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="kserve-container" containerID="cri-o://7d97acc9fa1328226355b49439994017c0ebc146c80128a92be8b9e5599cdaba" gracePeriod=30
Jul 24 21:36:31.971872 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:31.971693    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="kube-rbac-proxy" containerID="cri-o://44c017c134cf3b6916afb659ad97ae4ac54e99250f500d993fc7ce39007f5f55" gracePeriod=30
Jul 24 21:36:32.290029 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:32.289998    2580 generic.go:356] "Generic (PLEG): container finished" podID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerID="44c017c134cf3b6916afb659ad97ae4ac54e99250f500d993fc7ce39007f5f55" exitCode=2
Jul 24 21:36:32.290200 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:32.290068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" event={"ID":"328ae144-7d03-4fd8-93d4-fc2ffaf71c16","Type":"ContainerDied","Data":"44c017c134cf3b6916afb659ad97ae4ac54e99250f500d993fc7ce39007f5f55"}
Jul 24 21:36:33.297505 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.297471    2580 generic.go:356] "Generic (PLEG): container finished" podID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerID="7d97acc9fa1328226355b49439994017c0ebc146c80128a92be8b9e5599cdaba" exitCode=0
Jul 24 21:36:33.297840 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.297513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" event={"ID":"328ae144-7d03-4fd8-93d4-fc2ffaf71c16","Type":"ContainerDied","Data":"7d97acc9fa1328226355b49439994017c0ebc146c80128a92be8b9e5599cdaba"}
Jul 24 21:36:33.416252 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.416229    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:36:33.488124 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.488092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-proxy-tls\") pod \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") "
Jul 24 21:36:33.488302 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.488172    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kserve-provision-location\") pod \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") "
Jul 24 21:36:33.488375 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.488320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-cabundle-cert\") pod \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") "
Jul 24 21:36:33.488375 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.488370    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhlm7\" (UniqueName: \"kubernetes.io/projected/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-api-access-dhlm7\") pod \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") "
Jul 24 21:36:33.488530 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.488446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-rbac-proxy-sar-config\") pod \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\" (UID: \"328ae144-7d03-4fd8-93d4-fc2ffaf71c16\") "
Jul 24 21:36:33.488695 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.488669    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "328ae144-7d03-4fd8-93d4-fc2ffaf71c16" (UID: "328ae144-7d03-4fd8-93d4-fc2ffaf71c16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:36:33.488863 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.488842    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "328ae144-7d03-4fd8-93d4-fc2ffaf71c16" (UID: "328ae144-7d03-4fd8-93d4-fc2ffaf71c16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:36:33.490280 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.490260    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "328ae144-7d03-4fd8-93d4-fc2ffaf71c16" (UID: "328ae144-7d03-4fd8-93d4-fc2ffaf71c16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:36:33.490423 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.490373    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-api-access-dhlm7" (OuterVolumeSpecName: "kube-api-access-dhlm7") pod "328ae144-7d03-4fd8-93d4-fc2ffaf71c16" (UID: "328ae144-7d03-4fd8-93d4-fc2ffaf71c16"). InnerVolumeSpecName "kube-api-access-dhlm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:36:33.494574 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.494555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "328ae144-7d03-4fd8-93d4-fc2ffaf71c16" (UID: "328ae144-7d03-4fd8-93d4-fc2ffaf71c16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:36:33.589802 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.589770    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:36:33.589802 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.589800    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:36:33.589802 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.589810    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:36:33.590050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.589819    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:36:33.590050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:33.589828    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhlm7\" (UniqueName: \"kubernetes.io/projected/328ae144-7d03-4fd8-93d4-fc2ffaf71c16-kube-api-access-dhlm7\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:36:34.303199 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:34.303164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc" event={"ID":"328ae144-7d03-4fd8-93d4-fc2ffaf71c16","Type":"ContainerDied","Data":"e79be5f71e2cdc053d5a5c42233043ca1bc73c6f3eb4b48391e392be3f66cc92"}
Jul 24 21:36:34.303580 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:34.303212    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"
Jul 24 21:36:34.303580 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:34.303248    2580 scope.go:117] "RemoveContainer" containerID="44c017c134cf3b6916afb659ad97ae4ac54e99250f500d993fc7ce39007f5f55"
Jul 24 21:36:34.311880 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:34.311860    2580 scope.go:117] "RemoveContainer" containerID="7d97acc9fa1328226355b49439994017c0ebc146c80128a92be8b9e5599cdaba"
Jul 24 21:36:34.319809 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:34.319789    2580 scope.go:117] "RemoveContainer" containerID="f1f30d431198e0c5fac025bc50ac656f1e45a2a9c8fd39a44d34efc3019aad36"
Jul 24 21:36:34.331124 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:34.331101    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"]
Jul 24 21:36:34.332893 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:34.332872    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9g7nc"]
Jul 24 21:36:36.288034 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:36.288002    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" path="/var/lib/kubelet/pods/328ae144-7d03-4fd8-93d4-fc2ffaf71c16/volumes"
Jul 24 21:36:37.251644 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.251610    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"]
Jul 24 21:36:37.251980 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.251968    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="storage-initializer"
Jul 24 21:36:37.252038 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.251982    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="storage-initializer"
Jul 24 21:36:37.252038 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.251996    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="kserve-container"
Jul 24 21:36:37.252038 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.252002    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="kserve-container"
Jul 24 21:36:37.252038 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.252023    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="kube-rbac-proxy"
Jul 24 21:36:37.252038 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.252029    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="kube-rbac-proxy"
Jul 24 21:36:37.252204 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.252080    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="kube-rbac-proxy"
Jul 24 21:36:37.252204 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.252092    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="328ae144-7d03-4fd8-93d4-fc2ffaf71c16" containerName="kserve-container"
Jul 24 21:36:37.255452 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.255439    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.265477 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.265452    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:36:37.265477 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.265486    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:36:37.265671 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.265461    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:36:37.265671 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.265534    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 24 21:36:37.265671 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.265601    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 24 21:36:37.266477 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.266458    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:36:37.266524 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.266498    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:36:37.291737 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.291711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"]
Jul 24 21:36:37.320420 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.320374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-api-access-hkc5w\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.320585 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.320442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.320585 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.320479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.320585 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.320495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.320585 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.320516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8556d369-a6bc-4b01-94a8-acd73c0bbe12-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.421610 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.421573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-api-access-hkc5w\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.421807 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.421629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.421807 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.421652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.421807 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.421678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.421807 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.421707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8556d369-a6bc-4b01-94a8-acd73c0bbe12-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.422022 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:36:37.421852    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 24 21:36:37.422022 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:36:37.421943    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8556d369-a6bc-4b01-94a8-acd73c0bbe12-proxy-tls podName:8556d369-a6bc-4b01-94a8-acd73c0bbe12 nodeName:}" failed. No retries permitted until 2026-07-24 21:36:37.921922332 +0000 UTC m=+1272.260764087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8556d369-a6bc-4b01-94a8-acd73c0bbe12-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" (UID: "8556d369-a6bc-4b01-94a8-acd73c0bbe12") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 24 21:36:37.422145 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.422125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.422411 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.422373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.422600 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.422581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.433147 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.433118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-api-access-hkc5w\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.927279 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.927233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8556d369-a6bc-4b01-94a8-acd73c0bbe12-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:37.929800 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:37.929780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8556d369-a6bc-4b01-94a8-acd73c0bbe12-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:38.165575 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:38.165530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:38.305123 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:38.305097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"]
Jul 24 21:36:38.307087 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:36:38.307058    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8556d369_a6bc_4b01_94a8_acd73c0bbe12.slice/crio-afc905045a94f23af54de38f8b02051d186c12c12ac1ed8419ddf32f4a762174 WatchSource:0}: Error finding container afc905045a94f23af54de38f8b02051d186c12c12ac1ed8419ddf32f4a762174: Status 404 returned error can't find the container with id afc905045a94f23af54de38f8b02051d186c12c12ac1ed8419ddf32f4a762174
Jul 24 21:36:38.324198 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:38.324168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" event={"ID":"8556d369-a6bc-4b01-94a8-acd73c0bbe12","Type":"ContainerStarted","Data":"afc905045a94f23af54de38f8b02051d186c12c12ac1ed8419ddf32f4a762174"}
Jul 24 21:36:39.331033 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:39.331002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" event={"ID":"8556d369-a6bc-4b01-94a8-acd73c0bbe12","Type":"ContainerStarted","Data":"b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea"}
Jul 24 21:36:43.351796 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:43.351764    2580 generic.go:356] "Generic (PLEG): container finished" podID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerID="b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea" exitCode=0
Jul 24 21:36:43.352175 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:43.351842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" event={"ID":"8556d369-a6bc-4b01-94a8-acd73c0bbe12","Type":"ContainerDied","Data":"b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea"}
Jul 24 21:36:44.358770 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:44.358730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" event={"ID":"8556d369-a6bc-4b01-94a8-acd73c0bbe12","Type":"ContainerStarted","Data":"e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4"}
Jul 24 21:36:44.359185 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:44.358803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" event={"ID":"8556d369-a6bc-4b01-94a8-acd73c0bbe12","Type":"ContainerStarted","Data":"eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5"}
Jul 24 21:36:44.359185 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:44.359154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:44.381946 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:44.381886    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" podStartSLOduration=2.577887615 podStartE2EDuration="7.38186371s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.803976095s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:36:44.381177182 +0000 UTC m=+1278.720018959" watchObservedRunningTime="2026-07-24 21:36:44.38186371 +0000 UTC m=+1278.720705492"
Jul 24 21:36:45.363728 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:45.363697    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:36:51.373223 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:36:51.373193    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:38:17.286173 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:17.286086    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:38:28.322643 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:28.322583    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 21:38:34.267713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:34.267674    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"]
Jul 24 21:38:34.268261 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:34.267979    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="kserve-container" containerID="cri-o://eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5" gracePeriod=30
Jul 24 21:38:34.268261 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:34.268050    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="kube-rbac-proxy" containerID="cri-o://e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4" gracePeriod=30
Jul 24 21:38:34.865600 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:34.865567    2580 generic.go:356] "Generic (PLEG): container finished" podID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerID="e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4" exitCode=2
Jul 24 21:38:34.865816 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:34.865648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" event={"ID":"8556d369-a6bc-4b01-94a8-acd73c0bbe12","Type":"ContainerDied","Data":"e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4"}
Jul 24 21:38:35.708187 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.708160    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:38:35.853742 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.853659    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8556d369-a6bc-4b01-94a8-acd73c0bbe12-proxy-tls\") pod \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") "
Jul 24 21:38:35.853742 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.853739    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kserve-provision-location\") pod \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") "
Jul 24 21:38:35.853970 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.853765    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-rbac-proxy-sar-config\") pod \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") "
Jul 24 21:38:35.853970 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.853792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-cabundle-cert\") pod \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") "
Jul 24 21:38:35.853970 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.853844    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-api-access-hkc5w\") pod \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\" (UID: \"8556d369-a6bc-4b01-94a8-acd73c0bbe12\") "
Jul 24 21:38:35.854166 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.854144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8556d369-a6bc-4b01-94a8-acd73c0bbe12" (UID: "8556d369-a6bc-4b01-94a8-acd73c0bbe12"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:38:35.854226 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.854164    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8556d369-a6bc-4b01-94a8-acd73c0bbe12" (UID: "8556d369-a6bc-4b01-94a8-acd73c0bbe12"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:38:35.855917 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.855890    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8556d369-a6bc-4b01-94a8-acd73c0bbe12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8556d369-a6bc-4b01-94a8-acd73c0bbe12" (UID: "8556d369-a6bc-4b01-94a8-acd73c0bbe12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:38:35.856017 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.856002    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-api-access-hkc5w" (OuterVolumeSpecName: "kube-api-access-hkc5w") pod "8556d369-a6bc-4b01-94a8-acd73c0bbe12" (UID: "8556d369-a6bc-4b01-94a8-acd73c0bbe12"). InnerVolumeSpecName "kube-api-access-hkc5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:38:35.858569 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.858547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8556d369-a6bc-4b01-94a8-acd73c0bbe12" (UID: "8556d369-a6bc-4b01-94a8-acd73c0bbe12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:38:35.872258 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.872231    2580 generic.go:356] "Generic (PLEG): container finished" podID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerID="eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5" exitCode=0
Jul 24 21:38:35.872355 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.872300    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"
Jul 24 21:38:35.872355 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.872313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" event={"ID":"8556d369-a6bc-4b01-94a8-acd73c0bbe12","Type":"ContainerDied","Data":"eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5"}
Jul 24 21:38:35.872461 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.872373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4" event={"ID":"8556d369-a6bc-4b01-94a8-acd73c0bbe12","Type":"ContainerDied","Data":"afc905045a94f23af54de38f8b02051d186c12c12ac1ed8419ddf32f4a762174"}
Jul 24 21:38:35.872461 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.872416    2580 scope.go:117] "RemoveContainer" containerID="e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4"
Jul 24 21:38:35.881092 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.881075    2580 scope.go:117] "RemoveContainer" containerID="eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5"
Jul 24 21:38:35.888619 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.888603    2580 scope.go:117] "RemoveContainer" containerID="b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea"
Jul 24 21:38:35.895854 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.895835    2580 scope.go:117] "RemoveContainer" containerID="e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4"
Jul 24 21:38:35.896109 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:38:35.896091    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4\": container with ID starting with e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4 not found: ID does not exist" containerID="e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4"
Jul 24 21:38:35.896180 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.896123    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4"} err="failed to get container status \"e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4\": rpc error: code = NotFound desc = could not find container \"e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4\": container with ID starting with e08c7a1cef064baa7980f0eb136e326985d9af83f7fa8370ca1825fd584f49b4 not found: ID does not exist"
Jul 24 21:38:35.896180 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.896148    2580 scope.go:117] "RemoveContainer" containerID="eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5"
Jul 24 21:38:35.896408 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:38:35.896370    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5\": container with ID starting with eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5 not found: ID does not exist" containerID="eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5"
Jul 24 21:38:35.896464 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.896433    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5"} err="failed to get container status \"eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5\": rpc error: code = NotFound desc = could not find container \"eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5\": container with ID starting with eeeac110c1334d24b427e9b86e5e39e47209ab2189a8b515b01789d6fa24a4f5 not found: ID does not exist"
Jul 24 21:38:35.896464 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.896456    2580 scope.go:117] "RemoveContainer" containerID="b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea"
Jul 24 21:38:35.896712 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:38:35.896693    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea\": container with ID starting with b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea not found: ID does not exist" containerID="b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea"
Jul 24 21:38:35.896758 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.896717    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea"} err="failed to get container status \"b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea\": rpc error: code = NotFound desc = could not find container \"b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea\": container with ID starting with b2d77e28507a09f87404e36ca0564e638b62eefda8d750f6a1ce5ac822133aea not found: ID does not exist"
Jul 24 21:38:35.910526 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.910496    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"]
Jul 24 21:38:35.918663 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.918637    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-s9gn4"]
Jul 24 21:38:35.954927 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.954900    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-api-access-hkc5w\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:38:35.954927 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.954926    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8556d369-a6bc-4b01-94a8-acd73c0bbe12-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:38:35.955110 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.954940    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:38:35.955110 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.954953    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:38:35.955110 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:35.954965    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8556d369-a6bc-4b01-94a8-acd73c0bbe12-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:38:36.291836 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:36.289106    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" path="/var/lib/kubelet/pods/8556d369-a6bc-4b01-94a8-acd73c0bbe12/volumes"
Jul 24 21:38:39.350581 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.350547    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"]
Jul 24 21:38:39.351006 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.350992    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="storage-initializer"
Jul 24 21:38:39.351057 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.351010    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="storage-initializer"
Jul 24 21:38:39.351057 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.351019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="kserve-container"
Jul 24 21:38:39.351057 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.351025    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="kserve-container"
Jul 24 21:38:39.351057 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.351033    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="kube-rbac-proxy"
Jul 24 21:38:39.351057 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.351038    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="kube-rbac-proxy"
Jul 24 21:38:39.351209 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.351095    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="kserve-container"
Jul 24 21:38:39.351209 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.351102    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8556d369-a6bc-4b01-94a8-acd73c0bbe12" containerName="kube-rbac-proxy"
Jul 24 21:38:39.356017 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.356001    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.358568 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.358540    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:38:39.358709 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.358540    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 24 21:38:39.358709 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.358638    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:38:39.358801 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.358783    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:38:39.359065 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.359049    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:38:39.359568 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.359550    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 24 21:38:39.359640 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.359584    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:38:39.368047 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.368026    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"]
Jul 24 21:38:39.488562 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.488522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.488757 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.488586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4feb9d3c-af1d-4537-90a1-d6936a799a96-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.488757 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.488659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.488757 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.488707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqhf\" (UniqueName: \"kubernetes.io/projected/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-api-access-7gqhf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.488889 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.488767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4feb9d3c-af1d-4537-90a1-d6936a799a96-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.590059 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.590024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqhf\" (UniqueName: \"kubernetes.io/projected/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-api-access-7gqhf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.590356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.590106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4feb9d3c-af1d-4537-90a1-d6936a799a96-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.590356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.590149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.590356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.590205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4feb9d3c-af1d-4537-90a1-d6936a799a96-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.590356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.590241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.590653 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.590585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4feb9d3c-af1d-4537-90a1-d6936a799a96-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.590889 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.590860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.590975 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.590935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.592787 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.592765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4feb9d3c-af1d-4537-90a1-d6936a799a96-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.600636 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.600579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqhf\" (UniqueName: \"kubernetes.io/projected/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-api-access-7gqhf\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.667080 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.667038    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:38:39.810639 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.810610    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"]
Jul 24 21:38:39.814132 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:38:39.814102    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4feb9d3c_af1d_4537_90a1_d6936a799a96.slice/crio-7a294191bad70dec153ee54083a97a5823cbb647ddcc6b593dcd36c90cbaf74c WatchSource:0}: Error finding container 7a294191bad70dec153ee54083a97a5823cbb647ddcc6b593dcd36c90cbaf74c: Status 404 returned error can't find the container with id 7a294191bad70dec153ee54083a97a5823cbb647ddcc6b593dcd36c90cbaf74c
Jul 24 21:38:39.892703 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.892664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" event={"ID":"4feb9d3c-af1d-4537-90a1-d6936a799a96","Type":"ContainerStarted","Data":"0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454"}
Jul 24 21:38:39.892871 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:38:39.892745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" event={"ID":"4feb9d3c-af1d-4537-90a1-d6936a799a96","Type":"ContainerStarted","Data":"7a294191bad70dec153ee54083a97a5823cbb647ddcc6b593dcd36c90cbaf74c"}
Jul 24 21:39:10.033944 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:39:10.033909    2580 generic.go:356] "Generic (PLEG): container finished" podID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerID="0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454" exitCode=0
Jul 24 21:39:10.034335 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:39:10.033984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" event={"ID":"4feb9d3c-af1d-4537-90a1-d6936a799a96","Type":"ContainerDied","Data":"0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454"}
Jul 24 21:39:11.042035 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:39:11.042004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" event={"ID":"4feb9d3c-af1d-4537-90a1-d6936a799a96","Type":"ContainerStarted","Data":"9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a"}
Jul 24 21:39:11.042619 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:39:11.042064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" event={"ID":"4feb9d3c-af1d-4537-90a1-d6936a799a96","Type":"ContainerStarted","Data":"5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f"}
Jul 24 21:39:11.042619 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:39:11.042302    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:39:11.064620 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:39:11.064571    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" podStartSLOduration=2.876729413 podStartE2EDuration="32.064553623s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.18782421s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:39:11.063680152 +0000 UTC m=+1425.402521940" watchObservedRunningTime="2026-07-24 21:39:11.064553623 +0000 UTC m=+1425.403395399"
Jul 24 21:39:12.047071 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:39:12.047034    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:39:18.057115 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:39:18.057035    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:40:26.206188 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:26.206159    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:40:26.206720 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:26.206159    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:40:41.286381 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:41.286340    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:40:52.323614 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:52.323561    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 21:40:55.847379 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:55.847346    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"]
Jul 24 21:40:55.847873 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:55.847610    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="kserve-container" containerID="cri-o://5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f" gracePeriod=30
Jul 24 21:40:55.847873 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:55.847645    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="kube-rbac-proxy" containerID="cri-o://9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a" gracePeriod=30
Jul 24 21:40:56.514564 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:56.514530    2580 generic.go:356] "Generic (PLEG): container finished" podID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerID="9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a" exitCode=2
Jul 24 21:40:56.514769 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:56.514603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" event={"ID":"4feb9d3c-af1d-4537-90a1-d6936a799a96","Type":"ContainerDied","Data":"9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a"}
Jul 24 21:40:57.291620 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.291595    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:40:57.437822 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.437741    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-cabundle-cert\") pod \"4feb9d3c-af1d-4537-90a1-d6936a799a96\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") "
Jul 24 21:40:57.437822 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.437782    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4feb9d3c-af1d-4537-90a1-d6936a799a96-kserve-provision-location\") pod \"4feb9d3c-af1d-4537-90a1-d6936a799a96\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") "
Jul 24 21:40:57.438048 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.437865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gqhf\" (UniqueName: \"kubernetes.io/projected/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-api-access-7gqhf\") pod \"4feb9d3c-af1d-4537-90a1-d6936a799a96\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") "
Jul 24 21:40:57.438048 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.437896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-rbac-proxy-sar-config\") pod \"4feb9d3c-af1d-4537-90a1-d6936a799a96\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") "
Jul 24 21:40:57.438048 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.437951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4feb9d3c-af1d-4537-90a1-d6936a799a96-proxy-tls\") pod \"4feb9d3c-af1d-4537-90a1-d6936a799a96\" (UID: \"4feb9d3c-af1d-4537-90a1-d6936a799a96\") "
Jul 24 21:40:57.438272 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.438121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4feb9d3c-af1d-4537-90a1-d6936a799a96" (UID: "4feb9d3c-af1d-4537-90a1-d6936a799a96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:40:57.438272 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.438220    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:40:57.438380 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.438337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4feb9d3c-af1d-4537-90a1-d6936a799a96" (UID: "4feb9d3c-af1d-4537-90a1-d6936a799a96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:40:57.440243 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.440218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4feb9d3c-af1d-4537-90a1-d6936a799a96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4feb9d3c-af1d-4537-90a1-d6936a799a96" (UID: "4feb9d3c-af1d-4537-90a1-d6936a799a96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:40:57.440243 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.440231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-api-access-7gqhf" (OuterVolumeSpecName: "kube-api-access-7gqhf") pod "4feb9d3c-af1d-4537-90a1-d6936a799a96" (UID: "4feb9d3c-af1d-4537-90a1-d6936a799a96"). InnerVolumeSpecName "kube-api-access-7gqhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:40:57.443136 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.443108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4feb9d3c-af1d-4537-90a1-d6936a799a96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4feb9d3c-af1d-4537-90a1-d6936a799a96" (UID: "4feb9d3c-af1d-4537-90a1-d6936a799a96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:40:57.520724 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.520693    2580 generic.go:356] "Generic (PLEG): container finished" podID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerID="5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f" exitCode=0
Jul 24 21:40:57.520907 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.520757    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"
Jul 24 21:40:57.520907 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.520782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" event={"ID":"4feb9d3c-af1d-4537-90a1-d6936a799a96","Type":"ContainerDied","Data":"5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f"}
Jul 24 21:40:57.520907 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.520857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq" event={"ID":"4feb9d3c-af1d-4537-90a1-d6936a799a96","Type":"ContainerDied","Data":"7a294191bad70dec153ee54083a97a5823cbb647ddcc6b593dcd36c90cbaf74c"}
Jul 24 21:40:57.520907 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.520896    2580 scope.go:117] "RemoveContainer" containerID="9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a"
Jul 24 21:40:57.530204 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.530187    2580 scope.go:117] "RemoveContainer" containerID="5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f"
Jul 24 21:40:57.537978 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.537958    2580 scope.go:117] "RemoveContainer" containerID="0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454"
Jul 24 21:40:57.538633 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.538614    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqhf\" (UniqueName: \"kubernetes.io/projected/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-api-access-7gqhf\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:40:57.538710 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.538636    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4feb9d3c-af1d-4537-90a1-d6936a799a96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:40:57.538710 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.538650    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4feb9d3c-af1d-4537-90a1-d6936a799a96-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:40:57.538710 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.538664    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4feb9d3c-af1d-4537-90a1-d6936a799a96-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:40:57.546058 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.546040    2580 scope.go:117] "RemoveContainer" containerID="9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a"
Jul 24 21:40:57.546311 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:40:57.546291    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a\": container with ID starting with 9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a not found: ID does not exist" containerID="9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a"
Jul 24 21:40:57.546371 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.546320    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a"} err="failed to get container status \"9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a\": rpc error: code = NotFound desc = could not find container \"9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a\": container with ID starting with 9f1baa86753d4f62e1e409f10c05478f77830561e7488f188335a61a3a68297a not found: ID does not exist"
Jul 24 21:40:57.546371 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.546331    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"]
Jul 24 21:40:57.546371 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.546339    2580 scope.go:117] "RemoveContainer" containerID="5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f"
Jul 24 21:40:57.546673 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:40:57.546658    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f\": container with ID starting with 5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f not found: ID does not exist" containerID="5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f"
Jul 24 21:40:57.546713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.546680    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f"} err="failed to get container status \"5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f\": rpc error: code = NotFound desc = could not find container \"5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f\": container with ID starting with 5b86f6d6f44f8a15e87f1aa5ff1c0a307b414445d19ade145872446a6d9a8a2f not found: ID does not exist"
Jul 24 21:40:57.546713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.546696    2580 scope.go:117] "RemoveContainer" containerID="0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454"
Jul 24 21:40:57.546945 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:40:57.546928    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454\": container with ID starting with 0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454 not found: ID does not exist" containerID="0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454"
Jul 24 21:40:57.546986 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.546953    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454"} err="failed to get container status \"0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454\": rpc error: code = NotFound desc = could not find container \"0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454\": container with ID starting with 0340b106c84c547355b6b28696eb7b1f02261210c4f81b4da9fb0b97210ad454 not found: ID does not exist"
Jul 24 21:40:57.552062 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:57.552042    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9n4nq"]
Jul 24 21:40:58.287182 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:40:58.287141    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" path="/var/lib/kubelet/pods/4feb9d3c-af1d-4537-90a1-d6936a799a96/volumes"
Jul 24 21:41:01.020343 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.020310    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"]
Jul 24 21:41:01.020788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.020728    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="kube-rbac-proxy"
Jul 24 21:41:01.020788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.020740    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="kube-rbac-proxy"
Jul 24 21:41:01.020788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.020750    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="storage-initializer"
Jul 24 21:41:01.020788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.020755    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="storage-initializer"
Jul 24 21:41:01.020788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.020763    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="kserve-container"
Jul 24 21:41:01.020788 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.020769    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="kserve-container"
Jul 24 21:41:01.021071 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.020823    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="kube-rbac-proxy"
Jul 24 21:41:01.021071 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.020834    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4feb9d3c-af1d-4537-90a1-d6936a799a96" containerName="kserve-container"
Jul 24 21:41:01.025455 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.025435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.028264 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.028239    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:41:01.028746 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.028726    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:41:01.028871 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.028747    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:41:01.028871 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.028746    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 24 21:41:01.029024 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.028748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:41:01.029024 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.028748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 24 21:41:01.029024 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.028750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:41:01.035066 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.035045    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"]
Jul 24 21:41:01.059156 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.059123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.059156 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.059157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.059357 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.059180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a996e53-8807-4690-b662-5c250d52ccbd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.059357 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.059223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264ww\" (UniqueName: \"kubernetes.io/projected/1a996e53-8807-4690-b662-5c250d52ccbd-kube-api-access-264ww\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.059357 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.059255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a996e53-8807-4690-b662-5c250d52ccbd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.160406 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.160350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a996e53-8807-4690-b662-5c250d52ccbd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.160605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.160465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.160605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.160487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.160605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.160510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a996e53-8807-4690-b662-5c250d52ccbd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.160605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.160535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264ww\" (UniqueName: \"kubernetes.io/projected/1a996e53-8807-4690-b662-5c250d52ccbd-kube-api-access-264ww\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.160833 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.160804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a996e53-8807-4690-b662-5c250d52ccbd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.161084 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.161063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.161173 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.161127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.163068 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.163051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a996e53-8807-4690-b662-5c250d52ccbd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.171719 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.171697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-264ww\" (UniqueName: \"kubernetes.io/projected/1a996e53-8807-4690-b662-5c250d52ccbd-kube-api-access-264ww\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.336704 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.336601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:01.476172 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.476150    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"]
Jul 24 21:41:01.478327 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:41:01.478301    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a996e53_8807_4690_b662_5c250d52ccbd.slice/crio-0a058211966d096ce78c874e608bff4a2041e32bc58d5be28c9a49c466193f9c WatchSource:0}: Error finding container 0a058211966d096ce78c874e608bff4a2041e32bc58d5be28c9a49c466193f9c: Status 404 returned error can't find the container with id 0a058211966d096ce78c874e608bff4a2041e32bc58d5be28c9a49c466193f9c
Jul 24 21:41:01.540488 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.540454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" event={"ID":"1a996e53-8807-4690-b662-5c250d52ccbd","Type":"ContainerStarted","Data":"5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960"}
Jul 24 21:41:01.540615 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:01.540527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" event={"ID":"1a996e53-8807-4690-b662-5c250d52ccbd","Type":"ContainerStarted","Data":"0a058211966d096ce78c874e608bff4a2041e32bc58d5be28c9a49c466193f9c"}
Jul 24 21:41:11.591087 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:11.591051    2580 generic.go:356] "Generic (PLEG): container finished" podID="1a996e53-8807-4690-b662-5c250d52ccbd" containerID="5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960" exitCode=0
Jul 24 21:41:11.591495 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:11.591120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" event={"ID":"1a996e53-8807-4690-b662-5c250d52ccbd","Type":"ContainerDied","Data":"5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960"}
Jul 24 21:41:12.599019 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:12.598983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" event={"ID":"1a996e53-8807-4690-b662-5c250d52ccbd","Type":"ContainerStarted","Data":"7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34"}
Jul 24 21:41:12.599513 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:12.599053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" event={"ID":"1a996e53-8807-4690-b662-5c250d52ccbd","Type":"ContainerStarted","Data":"e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e"}
Jul 24 21:41:12.599513 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:12.599274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:12.619448 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:12.619369    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" podStartSLOduration=3.181480924 podStartE2EDuration="12.619346955s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.437866031s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:41:12.618211045 +0000 UTC m=+1546.957052821" watchObservedRunningTime="2026-07-24 21:41:12.619346955 +0000 UTC m=+1546.958188735"
Jul 24 21:41:13.603947 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:13.603916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:13.605257 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:13.605228    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:41:14.608887 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:14.608853    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:41:19.613579 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:19.613550    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:19.614229 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:19.614199    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:41:29.615141 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:29.615106    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:43.508170 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:43.508136    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"]
Jul 24 21:41:43.508582 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:43.508509    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kserve-container" containerID="cri-o://e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e" gracePeriod=30
Jul 24 21:41:43.508648 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:43.508562    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kube-rbac-proxy" containerID="cri-o://7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34" gracePeriod=30
Jul 24 21:41:43.745963 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:43.745926    2580 generic.go:356] "Generic (PLEG): container finished" podID="1a996e53-8807-4690-b662-5c250d52ccbd" containerID="7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34" exitCode=2
Jul 24 21:41:43.746142 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:43.745998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" event={"ID":"1a996e53-8807-4690-b662-5c250d52ccbd","Type":"ContainerDied","Data":"7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34"}
Jul 24 21:41:44.609945 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:44.609908    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 24 21:41:45.269375 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.269348    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:45.360112 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.360024    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a996e53-8807-4690-b662-5c250d52ccbd-proxy-tls\") pod \"1a996e53-8807-4690-b662-5c250d52ccbd\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") "
Jul 24 21:41:45.360112 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.360070    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-cabundle-cert\") pod \"1a996e53-8807-4690-b662-5c250d52ccbd\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") "
Jul 24 21:41:45.360112 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.360109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-264ww\" (UniqueName: \"kubernetes.io/projected/1a996e53-8807-4690-b662-5c250d52ccbd-kube-api-access-264ww\") pod \"1a996e53-8807-4690-b662-5c250d52ccbd\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") "
Jul 24 21:41:45.360415 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.360133    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-kube-rbac-proxy-sar-config\") pod \"1a996e53-8807-4690-b662-5c250d52ccbd\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") "
Jul 24 21:41:45.360415 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.360163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a996e53-8807-4690-b662-5c250d52ccbd-kserve-provision-location\") pod \"1a996e53-8807-4690-b662-5c250d52ccbd\" (UID: \"1a996e53-8807-4690-b662-5c250d52ccbd\") "
Jul 24 21:41:45.360566 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.360542    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a996e53-8807-4690-b662-5c250d52ccbd" (UID: "1a996e53-8807-4690-b662-5c250d52ccbd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:41:45.360623 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.360540    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a996e53-8807-4690-b662-5c250d52ccbd" (UID: "1a996e53-8807-4690-b662-5c250d52ccbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:41:45.361164 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.361137    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a996e53-8807-4690-b662-5c250d52ccbd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a996e53-8807-4690-b662-5c250d52ccbd" (UID: "1a996e53-8807-4690-b662-5c250d52ccbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:41:45.362288 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.362264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a996e53-8807-4690-b662-5c250d52ccbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a996e53-8807-4690-b662-5c250d52ccbd" (UID: "1a996e53-8807-4690-b662-5c250d52ccbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:41:45.362427 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.362365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a996e53-8807-4690-b662-5c250d52ccbd-kube-api-access-264ww" (OuterVolumeSpecName: "kube-api-access-264ww") pod "1a996e53-8807-4690-b662-5c250d52ccbd" (UID: "1a996e53-8807-4690-b662-5c250d52ccbd"). InnerVolumeSpecName "kube-api-access-264ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:41:45.461422 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.461351    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:41:45.461422 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.461383    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-264ww\" (UniqueName: \"kubernetes.io/projected/1a996e53-8807-4690-b662-5c250d52ccbd-kube-api-access-264ww\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:41:45.461422 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.461431    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a996e53-8807-4690-b662-5c250d52ccbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:41:45.461673 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.461443    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a996e53-8807-4690-b662-5c250d52ccbd-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:41:45.461673 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.461457    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a996e53-8807-4690-b662-5c250d52ccbd-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:41:45.757919 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.757835    2580 generic.go:356] "Generic (PLEG): container finished" podID="1a996e53-8807-4690-b662-5c250d52ccbd" containerID="e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e" exitCode=0
Jul 24 21:41:45.757919 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.757901    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"
Jul 24 21:41:45.758377 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.757914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" event={"ID":"1a996e53-8807-4690-b662-5c250d52ccbd","Type":"ContainerDied","Data":"e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e"}
Jul 24 21:41:45.758377 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.757971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh" event={"ID":"1a996e53-8807-4690-b662-5c250d52ccbd","Type":"ContainerDied","Data":"0a058211966d096ce78c874e608bff4a2041e32bc58d5be28c9a49c466193f9c"}
Jul 24 21:41:45.758377 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.757998    2580 scope.go:117] "RemoveContainer" containerID="7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34"
Jul 24 21:41:45.767006 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.766981    2580 scope.go:117] "RemoveContainer" containerID="e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e"
Jul 24 21:41:45.775169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.775147    2580 scope.go:117] "RemoveContainer" containerID="5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960"
Jul 24 21:41:45.779763 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.779739    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"]
Jul 24 21:41:45.783838 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.783818    2580 scope.go:117] "RemoveContainer" containerID="7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34"
Jul 24 21:41:45.783898 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.783881    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cbkzh"]
Jul 24 21:41:45.784145 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:41:45.784127    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34\": container with ID starting with 7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34 not found: ID does not exist" containerID="7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34"
Jul 24 21:41:45.784189 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.784157    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34"} err="failed to get container status \"7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34\": rpc error: code = NotFound desc = could not find container \"7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34\": container with ID starting with 7af8dcd6cc5e8f50f102dfe7b550547019974e2eb2d09914770b7824b756ee34 not found: ID does not exist"
Jul 24 21:41:45.784189 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.784176    2580 scope.go:117] "RemoveContainer" containerID="e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e"
Jul 24 21:41:45.784477 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:41:45.784459    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e\": container with ID starting with e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e not found: ID does not exist" containerID="e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e"
Jul 24 21:41:45.784556 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.784482    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e"} err="failed to get container status \"e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e\": rpc error: code = NotFound desc = could not find container \"e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e\": container with ID starting with e939c948c27059a16d5f8b0008b51dac0bbd4b010848a2537c4fc200d7445a2e not found: ID does not exist"
Jul 24 21:41:45.784556 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.784499    2580 scope.go:117] "RemoveContainer" containerID="5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960"
Jul 24 21:41:45.784727 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:41:45.784710    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960\": container with ID starting with 5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960 not found: ID does not exist" containerID="5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960"
Jul 24 21:41:45.784767 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:45.784732    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960"} err="failed to get container status \"5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960\": rpc error: code = NotFound desc = could not find container \"5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960\": container with ID starting with 5605be8590518eafffdfaed604f34d394f68f796ca13a7f4c409a4fa8e6ff960 not found: ID does not exist"
Jul 24 21:41:46.286780 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:46.286738    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" path="/var/lib/kubelet/pods/1a996e53-8807-4690-b662-5c250d52ccbd/volumes"
Jul 24 21:41:48.681258 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.681223    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"]
Jul 24 21:41:48.681659 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.681568    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="storage-initializer"
Jul 24 21:41:48.681659 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.681580    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="storage-initializer"
Jul 24 21:41:48.681659 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.681602    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kserve-container"
Jul 24 21:41:48.681659 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.681608    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kserve-container"
Jul 24 21:41:48.681659 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.681617    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kube-rbac-proxy"
Jul 24 21:41:48.681659 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.681622    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kube-rbac-proxy"
Jul 24 21:41:48.681853 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.681679    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kserve-container"
Jul 24 21:41:48.681853 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.681686    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a996e53-8807-4690-b662-5c250d52ccbd" containerName="kube-rbac-proxy"
Jul 24 21:41:48.686597 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.686578    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.689130 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.689102    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:41:48.689130 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.689102    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 24 21:41:48.689356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.689134    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 24 21:41:48.689356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.689234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:41:48.689356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.689260    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:41:48.689356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.689282    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:41:48.690097 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.690079    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:41:48.694687 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.694663    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"]
Jul 24 21:41:48.782384 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.782352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.782612 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.782433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbncr\" (UniqueName: \"kubernetes.io/projected/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-api-access-mbncr\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.782612 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.782485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.782612 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.782524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0bc9798-580a-4853-92e9-68e56bbeb7fc-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.782612 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.782562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.883141 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.883109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.883356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.883163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0bc9798-580a-4853-92e9-68e56bbeb7fc-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.883356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.883207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.883356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.883264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.883356 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.883304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbncr\" (UniqueName: \"kubernetes.io/projected/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-api-access-mbncr\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.883628 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:41:48.883363    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 24 21:41:48.883628 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:41:48.883471    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bc9798-580a-4853-92e9-68e56bbeb7fc-proxy-tls podName:d0bc9798-580a-4853-92e9-68e56bbeb7fc nodeName:}" failed. No retries permitted until 2026-07-24 21:41:49.383448629 +0000 UTC m=+1583.722290391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d0bc9798-580a-4853-92e9-68e56bbeb7fc-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" (UID: "d0bc9798-580a-4853-92e9-68e56bbeb7fc") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 24 21:41:48.883748 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.883681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.883883 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.883866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.883917 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.883868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:48.892197 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:48.892175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbncr\" (UniqueName: \"kubernetes.io/projected/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-api-access-mbncr\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:49.387232 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:49.387199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0bc9798-580a-4853-92e9-68e56bbeb7fc-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:49.389870 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:49.389838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0bc9798-580a-4853-92e9-68e56bbeb7fc-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:49.597613 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:49.597576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:49.741476 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:49.741447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"]
Jul 24 21:41:49.744504 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:41:49.744469    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bc9798_580a_4853_92e9_68e56bbeb7fc.slice/crio-be1e5fc938c14937c51b601a3d0afc0ac34a01c7a88fa356c7e47dac828e59bb WatchSource:0}: Error finding container be1e5fc938c14937c51b601a3d0afc0ac34a01c7a88fa356c7e47dac828e59bb: Status 404 returned error can't find the container with id be1e5fc938c14937c51b601a3d0afc0ac34a01c7a88fa356c7e47dac828e59bb
Jul 24 21:41:49.780089 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:49.780061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" event={"ID":"d0bc9798-580a-4853-92e9-68e56bbeb7fc","Type":"ContainerStarted","Data":"be1e5fc938c14937c51b601a3d0afc0ac34a01c7a88fa356c7e47dac828e59bb"}
Jul 24 21:41:50.786850 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:50.786810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" event={"ID":"d0bc9798-580a-4853-92e9-68e56bbeb7fc","Type":"ContainerStarted","Data":"c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0"}
Jul 24 21:41:54.808013 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:54.807981    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerID="c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0" exitCode=0
Jul 24 21:41:54.808410 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:54.808053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" event={"ID":"d0bc9798-580a-4853-92e9-68e56bbeb7fc","Type":"ContainerDied","Data":"c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0"}
Jul 24 21:41:55.814919 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:55.814878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" event={"ID":"d0bc9798-580a-4853-92e9-68e56bbeb7fc","Type":"ContainerStarted","Data":"e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef"}
Jul 24 21:41:55.815327 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:55.814940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" event={"ID":"d0bc9798-580a-4853-92e9-68e56bbeb7fc","Type":"ContainerStarted","Data":"02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c"}
Jul 24 21:41:55.815327 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:55.815217    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:55.815442 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:55.815408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:41:55.816645 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:55.816614    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:41:55.836285 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:55.836236    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" podStartSLOduration=3.3513518429999998 podStartE2EDuration="7.836220896s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.484869053s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:41:55.834262481 +0000 UTC m=+1590.173104267" watchObservedRunningTime="2026-07-24 21:41:55.836220896 +0000 UTC m=+1590.175062746"
Jul 24 21:41:56.819336 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:41:56.819295    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:42:01.823526 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:01.823501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:42:01.824081 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:01.824035    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:42:11.824561 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:11.824533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:42:23.907535 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:23.907453    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"]
Jul 24 21:42:23.907965 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:23.907708    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kserve-container" containerID="cri-o://02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c" gracePeriod=30
Jul 24 21:42:23.907965 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:23.907733    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kube-rbac-proxy" containerID="cri-o://e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef" gracePeriod=30
Jul 24 21:42:24.956929 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:24.956895    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerID="e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef" exitCode=2
Jul 24 21:42:24.957306 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:24.956975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" event={"ID":"d0bc9798-580a-4853-92e9-68e56bbeb7fc","Type":"ContainerDied","Data":"e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef"}
Jul 24 21:42:25.255285 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.255259    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:42:25.406972 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.406935    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbncr\" (UniqueName: \"kubernetes.io/projected/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-api-access-mbncr\") pod \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") "
Jul 24 21:42:25.407136 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.407015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-rbac-proxy-sar-config\") pod \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") "
Jul 24 21:42:25.407136 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.407055    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kserve-provision-location\") pod \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") "
Jul 24 21:42:25.407136 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.407105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-cabundle-cert\") pod \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") "
Jul 24 21:42:25.407323 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.407143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0bc9798-580a-4853-92e9-68e56bbeb7fc-proxy-tls\") pod \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\" (UID: \"d0bc9798-580a-4853-92e9-68e56bbeb7fc\") "
Jul 24 21:42:25.407499 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.407468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0bc9798-580a-4853-92e9-68e56bbeb7fc" (UID: "d0bc9798-580a-4853-92e9-68e56bbeb7fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:42:25.407578 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.407508    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0bc9798-580a-4853-92e9-68e56bbeb7fc" (UID: "d0bc9798-580a-4853-92e9-68e56bbeb7fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:42:25.407969 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.407949    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0bc9798-580a-4853-92e9-68e56bbeb7fc" (UID: "d0bc9798-580a-4853-92e9-68e56bbeb7fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:42:25.409197 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.409175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-api-access-mbncr" (OuterVolumeSpecName: "kube-api-access-mbncr") pod "d0bc9798-580a-4853-92e9-68e56bbeb7fc" (UID: "d0bc9798-580a-4853-92e9-68e56bbeb7fc"). InnerVolumeSpecName "kube-api-access-mbncr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:42:25.409290 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.409270    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bc9798-580a-4853-92e9-68e56bbeb7fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0bc9798-580a-4853-92e9-68e56bbeb7fc" (UID: "d0bc9798-580a-4853-92e9-68e56bbeb7fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:42:25.508680 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.508596    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.508680 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.508623    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.508680 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.508632    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0bc9798-580a-4853-92e9-68e56bbeb7fc-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.508680 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.508641    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbncr\" (UniqueName: \"kubernetes.io/projected/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-api-access-mbncr\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.508680 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.508649    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0bc9798-580a-4853-92e9-68e56bbeb7fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.963105 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.963068    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerID="02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c" exitCode=0
Jul 24 21:42:25.963569 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.963113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" event={"ID":"d0bc9798-580a-4853-92e9-68e56bbeb7fc","Type":"ContainerDied","Data":"02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c"}
Jul 24 21:42:25.963569 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.963121    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"
Jul 24 21:42:25.963569 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.963179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2" event={"ID":"d0bc9798-580a-4853-92e9-68e56bbeb7fc","Type":"ContainerDied","Data":"be1e5fc938c14937c51b601a3d0afc0ac34a01c7a88fa356c7e47dac828e59bb"}
Jul 24 21:42:25.963569 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.963217    2580 scope.go:117] "RemoveContainer" containerID="e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef"
Jul 24 21:42:25.972366 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.972346    2580 scope.go:117] "RemoveContainer" containerID="02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c"
Jul 24 21:42:25.980631 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.980610    2580 scope.go:117] "RemoveContainer" containerID="c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0"
Jul 24 21:42:25.987682 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.987595    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"]
Jul 24 21:42:25.991541 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.991484    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-s22m2"]
Jul 24 21:42:25.992111 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.992025    2580 scope.go:117] "RemoveContainer" containerID="e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef"
Jul 24 21:42:25.992710 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:42:25.992688    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef\": container with ID starting with e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef not found: ID does not exist" containerID="e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef"
Jul 24 21:42:25.992789 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.992720    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef"} err="failed to get container status \"e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef\": rpc error: code = NotFound desc = could not find container \"e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef\": container with ID starting with e2ab3ed37d94cee61ba27914000962823f4987795cb844b8aaa63507a6b677ef not found: ID does not exist"
Jul 24 21:42:25.992789 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.992741    2580 scope.go:117] "RemoveContainer" containerID="02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c"
Jul 24 21:42:25.993077 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:42:25.993056    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c\": container with ID starting with 02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c not found: ID does not exist" containerID="02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c"
Jul 24 21:42:25.993158 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.993088    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c"} err="failed to get container status \"02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c\": rpc error: code = NotFound desc = could not find container \"02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c\": container with ID starting with 02081ac9b55debaee1095adb2818930e950894e9391950366d3d995ecabb2c8c not found: ID does not exist"
Jul 24 21:42:25.993158 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.993114    2580 scope.go:117] "RemoveContainer" containerID="c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0"
Jul 24 21:42:25.993462 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:42:25.993443    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0\": container with ID starting with c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0 not found: ID does not exist" containerID="c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0"
Jul 24 21:42:25.993545 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:25.993469    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0"} err="failed to get container status \"c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0\": rpc error: code = NotFound desc = could not find container \"c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0\": container with ID starting with c64a5054ad9ffcda43d3de3dd4c2c925ecae35733e6589bea690c30a8b9caec0 not found: ID does not exist"
Jul 24 21:42:26.286894 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:26.286863    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" path="/var/lib/kubelet/pods/d0bc9798-580a-4853-92e9-68e56bbeb7fc/volumes"
Jul 24 21:42:29.177046 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.177011    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"]
Jul 24 21:42:29.177446 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.177375    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kserve-container"
Jul 24 21:42:29.177446 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.177400    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kserve-container"
Jul 24 21:42:29.177446 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.177415    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="storage-initializer"
Jul 24 21:42:29.177446 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.177420    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="storage-initializer"
Jul 24 21:42:29.177446 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.177438    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kube-rbac-proxy"
Jul 24 21:42:29.177446 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.177444    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kube-rbac-proxy"
Jul 24 21:42:29.177651 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.177497    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kserve-container"
Jul 24 21:42:29.177651 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.177504    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bc9798-580a-4853-92e9-68e56bbeb7fc" containerName="kube-rbac-proxy"
Jul 24 21:42:29.182169 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.182150    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.188954 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.188934    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 24 21:42:29.189081 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.189041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 24 21:42:29.189081 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.189049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:42:29.189231 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.189183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:42:29.189231 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.189190    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:42:29.189329 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.189286    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:42:29.189863 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.189840    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:42:29.200087 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.200067    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"]
Jul 24 21:42:29.255078 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.255039    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"]
Jul 24 21:42:29.258777 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.258746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.266939 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.266919    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 24 21:42:29.277981 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.277958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"]
Jul 24 21:42:29.344133 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0105e160-ac97-4f97-9776-4338bb6996ba-proxy-tls\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.344133 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0105e160-ac97-4f97-9776-4338bb6996ba-kserve-provision-location\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.344341 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpjc\" (UniqueName: \"kubernetes.io/projected/0105e160-ac97-4f97-9776-4338bb6996ba-kube-api-access-9bpjc\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.344341 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294d0d0a-f179-4809-a90f-3e738a9c6160-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.344341 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.344341 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-cabundle-cert\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.344341 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpgw\" (UniqueName: \"kubernetes.io/projected/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-api-access-nvpgw\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.344341 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294d0d0a-f179-4809-a90f-3e738a9c6160-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.344561 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.344561 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.344461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.445617 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294d0d0a-f179-4809-a90f-3e738a9c6160-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.445617 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.445617 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-cabundle-cert\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.445890 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpgw\" (UniqueName: \"kubernetes.io/projected/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-api-access-nvpgw\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.445890 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294d0d0a-f179-4809-a90f-3e738a9c6160-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.445890 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.445890 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.445890 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0105e160-ac97-4f97-9776-4338bb6996ba-proxy-tls\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.445890 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0105e160-ac97-4f97-9776-4338bb6996ba-kserve-provision-location\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.446228 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.445931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpjc\" (UniqueName: \"kubernetes.io/projected/0105e160-ac97-4f97-9776-4338bb6996ba-kube-api-access-9bpjc\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.446489 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.446337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0105e160-ac97-4f97-9776-4338bb6996ba-kserve-provision-location\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.446489 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.446343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294d0d0a-f179-4809-a90f-3e738a9c6160-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.446489 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.446416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.446734 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.446622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-cabundle-cert\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.446809 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.446775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.446990 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.446969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.448108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.448086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294d0d0a-f179-4809-a90f-3e738a9c6160-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.448376 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.448360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0105e160-ac97-4f97-9776-4338bb6996ba-proxy-tls\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.458336 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.458315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpjc\" (UniqueName: \"kubernetes.io/projected/0105e160-ac97-4f97-9776-4338bb6996ba-kube-api-access-9bpjc\") pod \"isvc-canary-create-predictor-7f74cfd478-88wlx\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.465434 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.465412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpgw\" (UniqueName: \"kubernetes.io/projected/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-api-access-nvpgw\") pod \"isvc-canary-create-v2-predictor-75c7994655-s4p7r\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.492361 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.492324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:29.568253 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.568204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:29.637928 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.637901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"]
Jul 24 21:42:29.639687 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:42:29.639645    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0105e160_ac97_4f97_9776_4338bb6996ba.slice/crio-403b3c968b7be4cdcafb04508b4ccc64cf1b8e629592ca91a961d3f6836eaa7e WatchSource:0}: Error finding container 403b3c968b7be4cdcafb04508b4ccc64cf1b8e629592ca91a961d3f6836eaa7e: Status 404 returned error can't find the container with id 403b3c968b7be4cdcafb04508b4ccc64cf1b8e629592ca91a961d3f6836eaa7e
Jul 24 21:42:29.718534 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.718509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"]
Jul 24 21:42:29.721167 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:42:29.721143    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294d0d0a_f179_4809_a90f_3e738a9c6160.slice/crio-bf1e05089d1e9ff508847d4af2e276a732d56ec311eccaba8b5cf2da765a2d10 WatchSource:0}: Error finding container bf1e05089d1e9ff508847d4af2e276a732d56ec311eccaba8b5cf2da765a2d10: Status 404 returned error can't find the container with id bf1e05089d1e9ff508847d4af2e276a732d56ec311eccaba8b5cf2da765a2d10
Jul 24 21:42:29.984261 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.984165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" event={"ID":"294d0d0a-f179-4809-a90f-3e738a9c6160","Type":"ContainerStarted","Data":"c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d"}
Jul 24 21:42:29.984261 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.984246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" event={"ID":"294d0d0a-f179-4809-a90f-3e738a9c6160","Type":"ContainerStarted","Data":"bf1e05089d1e9ff508847d4af2e276a732d56ec311eccaba8b5cf2da765a2d10"}
Jul 24 21:42:29.986027 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.986001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" event={"ID":"0105e160-ac97-4f97-9776-4338bb6996ba","Type":"ContainerStarted","Data":"6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8"}
Jul 24 21:42:29.986155 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:29.986058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" event={"ID":"0105e160-ac97-4f97-9776-4338bb6996ba","Type":"ContainerStarted","Data":"403b3c968b7be4cdcafb04508b4ccc64cf1b8e629592ca91a961d3f6836eaa7e"}
Jul 24 21:42:34.007041 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:34.006994    2580 generic.go:356] "Generic (PLEG): container finished" podID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerID="c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d" exitCode=0
Jul 24 21:42:34.007533 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:34.007060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" event={"ID":"294d0d0a-f179-4809-a90f-3e738a9c6160","Type":"ContainerDied","Data":"c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d"}
Jul 24 21:42:34.008889 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:34.008861    2580 generic.go:356] "Generic (PLEG): container finished" podID="0105e160-ac97-4f97-9776-4338bb6996ba" containerID="6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8" exitCode=0
Jul 24 21:42:34.008995 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:34.008897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" event={"ID":"0105e160-ac97-4f97-9776-4338bb6996ba","Type":"ContainerDied","Data":"6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8"}
Jul 24 21:42:35.015880 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.015846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" event={"ID":"294d0d0a-f179-4809-a90f-3e738a9c6160","Type":"ContainerStarted","Data":"ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df"}
Jul 24 21:42:35.016342 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.015910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" event={"ID":"294d0d0a-f179-4809-a90f-3e738a9c6160","Type":"ContainerStarted","Data":"12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631"}
Jul 24 21:42:35.016342 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.016262    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:35.016492 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.016422    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:35.018201 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.018173    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:42:35.018423 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.018380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" event={"ID":"0105e160-ac97-4f97-9776-4338bb6996ba","Type":"ContainerStarted","Data":"453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673"}
Jul 24 21:42:35.018503 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.018464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" event={"ID":"0105e160-ac97-4f97-9776-4338bb6996ba","Type":"ContainerStarted","Data":"a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d"}
Jul 24 21:42:35.018681 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.018665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:35.018798 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.018701    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:35.019687 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.019642    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:42:35.037470 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.037418    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podStartSLOduration=2.357751088 podStartE2EDuration="6.037377559s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.679626471s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:42:35.0355181 +0000 UTC m=+1629.374359889" watchObservedRunningTime="2026-07-24 21:42:35.037377559 +0000 UTC m=+1629.376219334"
Jul 24 21:42:35.056899 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:35.056850    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podStartSLOduration=2.289618019 podStartE2EDuration="6.056830805s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767212786s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:42:35.055111001 +0000 UTC m=+1629.393952778" watchObservedRunningTime="2026-07-24 21:42:35.056830805 +0000 UTC m=+1629.395672580"
Jul 24 21:42:36.022789 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:36.022728    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:42:36.023205 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:36.022890    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:42:41.027093 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:41.027063    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:42:41.027568 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:41.027458    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:42:41.027568 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:41.027533    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:42:41.027940 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:41.027909    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:42:51.027837 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:51.027796    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:42:51.028251 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:42:51.027948    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:43:01.027687 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:01.027650    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:43:01.028106 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:01.027901    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:43:11.027611 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:11.027575    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:43:11.028088 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:11.027797    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:43:21.028237 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:21.028200    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:43:21.028705 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:21.028200    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:43:31.027828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:31.027791    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:43:31.028223 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:31.027930    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:43:41.028099 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:41.028064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:43:41.028526 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:41.028061    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:43:50.287713 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:50.287626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:43:51.028755 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:51.028724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:43:59.301217 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:59.301182    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"]
Jul 24 21:43:59.301726 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:59.301542    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" containerID="cri-o://a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d" gracePeriod=30
Jul 24 21:43:59.301726 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:59.301545    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kube-rbac-proxy" containerID="cri-o://453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673" gracePeriod=30
Jul 24 21:43:59.309898 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:59.309871    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"]
Jul 24 21:43:59.310153 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:59.310117    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" containerID="cri-o://12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631" gracePeriod=30
Jul 24 21:43:59.310272 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:59.310223    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kube-rbac-proxy" containerID="cri-o://ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df" gracePeriod=30
Jul 24 21:43:59.418034 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:59.418002    2580 generic.go:356] "Generic (PLEG): container finished" podID="0105e160-ac97-4f97-9776-4338bb6996ba" containerID="453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673" exitCode=2
Jul 24 21:43:59.418157 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:43:59.418071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" event={"ID":"0105e160-ac97-4f97-9776-4338bb6996ba","Type":"ContainerDied","Data":"453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673"}
Jul 24 21:44:00.282274 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:00.282228    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:44:00.425498 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:00.425466    2580 generic.go:356] "Generic (PLEG): container finished" podID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerID="ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df" exitCode=2
Jul 24 21:44:00.425839 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:00.425540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" event={"ID":"294d0d0a-f179-4809-a90f-3e738a9c6160","Type":"ContainerDied","Data":"ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df"}
Jul 24 21:44:01.023743 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:01.023703    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 24 21:44:01.023925 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:01.023712    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 24 21:44:01.028873 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:01.028849    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 24 21:44:04.182639 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.182607    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:44:04.185840 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.185817    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:44:04.203783 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.203749    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294d0d0a-f179-4809-a90f-3e738a9c6160-kserve-provision-location\") pod \"294d0d0a-f179-4809-a90f-3e738a9c6160\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") "
Jul 24 21:44:04.203958 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.203823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294d0d0a-f179-4809-a90f-3e738a9c6160-proxy-tls\") pod \"294d0d0a-f179-4809-a90f-3e738a9c6160\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") "
Jul 24 21:44:04.203958 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.203862    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-cabundle-cert\") pod \"0105e160-ac97-4f97-9776-4338bb6996ba\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") "
Jul 24 21:44:04.203958 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.203892    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-cabundle-cert\") pod \"294d0d0a-f179-4809-a90f-3e738a9c6160\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") "
Jul 24 21:44:04.203958 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.203921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0105e160-ac97-4f97-9776-4338bb6996ba-kserve-provision-location\") pod \"0105e160-ac97-4f97-9776-4338bb6996ba\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") "
Jul 24 21:44:04.204222 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.203972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-rbac-proxy-sar-config\") pod \"294d0d0a-f179-4809-a90f-3e738a9c6160\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") "
Jul 24 21:44:04.204222 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.204109    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294d0d0a-f179-4809-a90f-3e738a9c6160-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "294d0d0a-f179-4809-a90f-3e738a9c6160" (UID: "294d0d0a-f179-4809-a90f-3e738a9c6160"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:44:04.204515 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.204436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "294d0d0a-f179-4809-a90f-3e738a9c6160" (UID: "294d0d0a-f179-4809-a90f-3e738a9c6160"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:44:04.204515 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.204490    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0105e160-ac97-4f97-9776-4338bb6996ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0105e160-ac97-4f97-9776-4338bb6996ba" (UID: "0105e160-ac97-4f97-9776-4338bb6996ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:44:04.204733 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.204555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0105e160-ac97-4f97-9776-4338bb6996ba" (UID: "0105e160-ac97-4f97-9776-4338bb6996ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:44:04.204733 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.204613    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bpjc\" (UniqueName: \"kubernetes.io/projected/0105e160-ac97-4f97-9776-4338bb6996ba-kube-api-access-9bpjc\") pod \"0105e160-ac97-4f97-9776-4338bb6996ba\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") "
Jul 24 21:44:04.204733 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.204617    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "294d0d0a-f179-4809-a90f-3e738a9c6160" (UID: "294d0d0a-f179-4809-a90f-3e738a9c6160"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:44:04.204733 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.204662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvpgw\" (UniqueName: \"kubernetes.io/projected/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-api-access-nvpgw\") pod \"294d0d0a-f179-4809-a90f-3e738a9c6160\" (UID: \"294d0d0a-f179-4809-a90f-3e738a9c6160\") "
Jul 24 21:44:04.204733 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.204691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0105e160-ac97-4f97-9776-4338bb6996ba-proxy-tls\") pod \"0105e160-ac97-4f97-9776-4338bb6996ba\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") "
Jul 24 21:44:04.204733 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.204724    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-kube-rbac-proxy-sar-config\") pod \"0105e160-ac97-4f97-9776-4338bb6996ba\" (UID: \"0105e160-ac97-4f97-9776-4338bb6996ba\") "
Jul 24 21:44:04.205403 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.205358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0105e160-ac97-4f97-9776-4338bb6996ba" (UID: "0105e160-ac97-4f97-9776-4338bb6996ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:44:04.205533 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.205439    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294d0d0a-f179-4809-a90f-3e738a9c6160-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.205533 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.205457    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.205533 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.205471    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.205533 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.205482    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0105e160-ac97-4f97-9776-4338bb6996ba-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.205533 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.205496    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.207234 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.207199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294d0d0a-f179-4809-a90f-3e738a9c6160-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "294d0d0a-f179-4809-a90f-3e738a9c6160" (UID: "294d0d0a-f179-4809-a90f-3e738a9c6160"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:44:04.207583 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.207553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0105e160-ac97-4f97-9776-4338bb6996ba-kube-api-access-9bpjc" (OuterVolumeSpecName: "kube-api-access-9bpjc") pod "0105e160-ac97-4f97-9776-4338bb6996ba" (UID: "0105e160-ac97-4f97-9776-4338bb6996ba"). InnerVolumeSpecName "kube-api-access-9bpjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:44:04.207583 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.207562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-api-access-nvpgw" (OuterVolumeSpecName: "kube-api-access-nvpgw") pod "294d0d0a-f179-4809-a90f-3e738a9c6160" (UID: "294d0d0a-f179-4809-a90f-3e738a9c6160"). InnerVolumeSpecName "kube-api-access-nvpgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:44:04.207927 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.207894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0105e160-ac97-4f97-9776-4338bb6996ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0105e160-ac97-4f97-9776-4338bb6996ba" (UID: "0105e160-ac97-4f97-9776-4338bb6996ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:44:04.306734 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.306711    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294d0d0a-f179-4809-a90f-3e738a9c6160-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.306734 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.306734    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpjc\" (UniqueName: \"kubernetes.io/projected/0105e160-ac97-4f97-9776-4338bb6996ba-kube-api-access-9bpjc\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.306903 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.306745    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvpgw\" (UniqueName: \"kubernetes.io/projected/294d0d0a-f179-4809-a90f-3e738a9c6160-kube-api-access-nvpgw\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.306903 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.306754    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0105e160-ac97-4f97-9776-4338bb6996ba-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.306903 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.306772    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0105e160-ac97-4f97-9776-4338bb6996ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:44:04.399190 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399154    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"]
Jul 24 21:44:04.399523 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399510    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="storage-initializer"
Jul 24 21:44:04.399572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399525    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="storage-initializer"
Jul 24 21:44:04.399572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399540    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container"
Jul 24 21:44:04.399572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399546    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container"
Jul 24 21:44:04.399572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399554    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="storage-initializer"
Jul 24 21:44:04.399572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399561    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="storage-initializer"
Jul 24 21:44:04.399572 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399571    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kube-rbac-proxy"
Jul 24 21:44:04.399828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399577    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kube-rbac-proxy"
Jul 24 21:44:04.399828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399584    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kube-rbac-proxy"
Jul 24 21:44:04.399828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399589    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kube-rbac-proxy"
Jul 24 21:44:04.399828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399601    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container"
Jul 24 21:44:04.399828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399606    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container"
Jul 24 21:44:04.399828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399655    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kube-rbac-proxy"
Jul 24 21:44:04.399828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399663    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kube-rbac-proxy"
Jul 24 21:44:04.399828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399671    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerName="kserve-container"
Jul 24 21:44:04.399828 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.399678    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" containerName="kserve-container"
Jul 24 21:44:04.404369 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.404349    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"]
Jul 24 21:44:04.404515 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.404497    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.406810 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.406785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 24 21:44:04.406929 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.406872    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 24 21:44:04.407954 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.407936    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.409921 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.409903    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 24 21:44:04.413248 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.413227    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"]
Jul 24 21:44:04.416382 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.416362    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"]
Jul 24 21:44:04.447203 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.447169    2580 generic.go:356] "Generic (PLEG): container finished" podID="294d0d0a-f179-4809-a90f-3e738a9c6160" containerID="12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631" exitCode=0
Jul 24 21:44:04.447510 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.447294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" event={"ID":"294d0d0a-f179-4809-a90f-3e738a9c6160","Type":"ContainerDied","Data":"12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631"}
Jul 24 21:44:04.447510 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.447360    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"
Jul 24 21:44:04.447510 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.447410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r" event={"ID":"294d0d0a-f179-4809-a90f-3e738a9c6160","Type":"ContainerDied","Data":"bf1e05089d1e9ff508847d4af2e276a732d56ec311eccaba8b5cf2da765a2d10"}
Jul 24 21:44:04.447510 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.447477    2580 scope.go:117] "RemoveContainer" containerID="ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df"
Jul 24 21:44:04.449683 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.449660    2580 generic.go:356] "Generic (PLEG): container finished" podID="0105e160-ac97-4f97-9776-4338bb6996ba" containerID="a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d" exitCode=0
Jul 24 21:44:04.449797 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.449692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" event={"ID":"0105e160-ac97-4f97-9776-4338bb6996ba","Type":"ContainerDied","Data":"a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d"}
Jul 24 21:44:04.449797 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.449728    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"
Jul 24 21:44:04.449797 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.449743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx" event={"ID":"0105e160-ac97-4f97-9776-4338bb6996ba","Type":"ContainerDied","Data":"403b3c968b7be4cdcafb04508b4ccc64cf1b8e629592ca91a961d3f6836eaa7e"}
Jul 24 21:44:04.456693 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.456676    2580 scope.go:117] "RemoveContainer" containerID="12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631"
Jul 24 21:44:04.464281 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.464264    2580 scope.go:117] "RemoveContainer" containerID="c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d"
Jul 24 21:44:04.467790 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.467770    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"]
Jul 24 21:44:04.472042 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.472015    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-s4p7r"]
Jul 24 21:44:04.473241 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.473217    2580 scope.go:117] "RemoveContainer" containerID="ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df"
Jul 24 21:44:04.473552 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:44:04.473532    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df\": container with ID starting with ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df not found: ID does not exist" containerID="ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df"
Jul 24 21:44:04.473609 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.473562    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df"} err="failed to get container status \"ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df\": rpc error: code = NotFound desc = could not find container \"ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df\": container with ID starting with ed1fb76931de64159846990d98dd5be73f7d725a02f0378553538144bdce60df not found: ID does not exist"
Jul 24 21:44:04.473609 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.473580    2580 scope.go:117] "RemoveContainer" containerID="12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631"
Jul 24 21:44:04.473817 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:44:04.473799    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631\": container with ID starting with 12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631 not found: ID does not exist" containerID="12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631"
Jul 24 21:44:04.473886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.473827    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631"} err="failed to get container status \"12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631\": rpc error: code = NotFound desc = could not find container \"12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631\": container with ID starting with 12e0a78d3584ae877b3eb317b7cbacbb4e96ab6a567ff50942ed506a65c8d631 not found: ID does not exist"
Jul 24 21:44:04.473886 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.473851    2580 scope.go:117] "RemoveContainer" containerID="c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d"
Jul 24 21:44:04.474088 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:44:04.474068    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d\": container with ID starting with c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d not found: ID does not exist" containerID="c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d"
Jul 24 21:44:04.474123 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.474094    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d"} err="failed to get container status \"c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d\": rpc error: code = NotFound desc = could not find container \"c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d\": container with ID starting with c531a1b65d29c0e6c3374936dcae82440686e439116968c490204b43ac0b6c7d not found: ID does not exist"
Jul 24 21:44:04.474123 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.474109    2580 scope.go:117] "RemoveContainer" containerID="453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673"
Jul 24 21:44:04.482103 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.482078    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"]
Jul 24 21:44:04.482318 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.482303    2580 scope.go:117] "RemoveContainer" containerID="a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d"
Jul 24 21:44:04.486153 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.486131    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-88wlx"]
Jul 24 21:44:04.490471 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.490454    2580 scope.go:117] "RemoveContainer" containerID="6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8"
Jul 24 21:44:04.498148 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.498131    2580 scope.go:117] "RemoveContainer" containerID="453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673"
Jul 24 21:44:04.498433 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:44:04.498415    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673\": container with ID starting with 453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673 not found: ID does not exist" containerID="453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673"
Jul 24 21:44:04.498504 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.498440    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673"} err="failed to get container status \"453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673\": rpc error: code = NotFound desc = could not find container \"453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673\": container with ID starting with 453b1be0dc3d2fcf1ddc6d940f7950af310c878ee826e005ba1604afb4256673 not found: ID does not exist"
Jul 24 21:44:04.498504 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.498459    2580 scope.go:117] "RemoveContainer" containerID="a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d"
Jul 24 21:44:04.498703 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:44:04.498685    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d\": container with ID starting with a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d not found: ID does not exist" containerID="a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d"
Jul 24 21:44:04.498746 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.498709    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d"} err="failed to get container status \"a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d\": rpc error: code = NotFound desc = could not find container \"a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d\": container with ID starting with a3e1e708495d9fc24f697d6b98d6291c6e5187fd59ecc32074707bfd3204339d not found: ID does not exist"
Jul 24 21:44:04.498746 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.498724    2580 scope.go:117] "RemoveContainer" containerID="6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8"
Jul 24 21:44:04.498964 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:44:04.498949    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8\": container with ID starting with 6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8 not found: ID does not exist" containerID="6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8"
Jul 24 21:44:04.499005 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.498973    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8"} err="failed to get container status \"6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8\": rpc error: code = NotFound desc = could not find container \"6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8\": container with ID starting with 6222af82b1a9983c6e73d8615777b1ca9ca94a16750bc677bfe3e9932d9203a8 not found: ID does not exist"
Jul 24 21:44:04.508374 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85153d39-9664-4ecc-80db-d30444efdba3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.508479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b817a8c3-95c5-449a-91ba-3e421f8ed327-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.508479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85153d39-9664-4ecc-80db-d30444efdba3-proxy-tls\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.508479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.508579 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-cabundle-cert\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.508579 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.508579 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.508675 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnl6m\" (UniqueName: \"kubernetes.io/projected/85153d39-9664-4ecc-80db-d30444efdba3-kube-api-access-wnl6m\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.508675 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b817a8c3-95c5-449a-91ba-3e421f8ed327-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.508735 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.508680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hs44\" (UniqueName: \"kubernetes.io/projected/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-api-access-2hs44\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.609686 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.609635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnl6m\" (UniqueName: \"kubernetes.io/projected/85153d39-9664-4ecc-80db-d30444efdba3-kube-api-access-wnl6m\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.609686 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.609686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b817a8c3-95c5-449a-91ba-3e421f8ed327-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.609884 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.609801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hs44\" (UniqueName: \"kubernetes.io/projected/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-api-access-2hs44\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.609884 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.609870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85153d39-9664-4ecc-80db-d30444efdba3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.609956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.609891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b817a8c3-95c5-449a-91ba-3e421f8ed327-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.609956 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.609939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85153d39-9664-4ecc-80db-d30444efdba3-proxy-tls\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.610050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.609970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.610050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.609993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b817a8c3-95c5-449a-91ba-3e421f8ed327-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.610050 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.610024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-cabundle-cert\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.610202 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.610072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.610202 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.610177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.610336 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.610310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85153d39-9664-4ecc-80db-d30444efdba3-kserve-provision-location\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.610694 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.610670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.610787 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.610716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.610787 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.610776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-cabundle-cert\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.610862 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.610776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.612504 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.612489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b817a8c3-95c5-449a-91ba-3e421f8ed327-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.612866 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.612852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85153d39-9664-4ecc-80db-d30444efdba3-proxy-tls\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.618206 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.618181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnl6m\" (UniqueName: \"kubernetes.io/projected/85153d39-9664-4ecc-80db-d30444efdba3-kube-api-access-wnl6m\") pod \"isvc-canary-promote-predictor-74874cc84b-7b45l\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.618278 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.618223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hs44\" (UniqueName: \"kubernetes.io/projected/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-api-access-2hs44\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.717555 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.717475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:04.726566 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.726545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:04.865171 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.865145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"]
Jul 24 21:44:04.867121 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:44:04.867092    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85153d39_9664_4ecc_80db_d30444efdba3.slice/crio-e24f2f55c8a779b4c071b03fd65288a6d097c458cbe6b393f212e1ff9d71db7a WatchSource:0}: Error finding container e24f2f55c8a779b4c071b03fd65288a6d097c458cbe6b393f212e1ff9d71db7a: Status 404 returned error can't find the container with id e24f2f55c8a779b4c071b03fd65288a6d097c458cbe6b393f212e1ff9d71db7a
Jul 24 21:44:04.886030 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:04.886009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"]
Jul 24 21:44:04.889063 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:44:04.889038    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb817a8c3_95c5_449a_91ba_3e421f8ed327.slice/crio-a5de979c6ef6cc91f2789806fc583c0c44685cbc119fc98a68991e594634da8c WatchSource:0}: Error finding container a5de979c6ef6cc91f2789806fc583c0c44685cbc119fc98a68991e594634da8c: Status 404 returned error can't find the container with id a5de979c6ef6cc91f2789806fc583c0c44685cbc119fc98a68991e594634da8c
Jul 24 21:44:05.459545 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:05.459502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" event={"ID":"85153d39-9664-4ecc-80db-d30444efdba3","Type":"ContainerStarted","Data":"7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283"}
Jul 24 21:44:05.460017 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:05.459575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" event={"ID":"85153d39-9664-4ecc-80db-d30444efdba3","Type":"ContainerStarted","Data":"e24f2f55c8a779b4c071b03fd65288a6d097c458cbe6b393f212e1ff9d71db7a"}
Jul 24 21:44:05.461272 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:05.461244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" event={"ID":"b817a8c3-95c5-449a-91ba-3e421f8ed327","Type":"ContainerStarted","Data":"27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2"}
Jul 24 21:44:05.461418 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:05.461301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" event={"ID":"b817a8c3-95c5-449a-91ba-3e421f8ed327","Type":"ContainerStarted","Data":"a5de979c6ef6cc91f2789806fc583c0c44685cbc119fc98a68991e594634da8c"}
Jul 24 21:44:06.287663 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:06.287631    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0105e160-ac97-4f97-9776-4338bb6996ba" path="/var/lib/kubelet/pods/0105e160-ac97-4f97-9776-4338bb6996ba/volumes"
Jul 24 21:44:06.288138 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:06.288125    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="294d0d0a-f179-4809-a90f-3e738a9c6160" path="/var/lib/kubelet/pods/294d0d0a-f179-4809-a90f-3e738a9c6160/volumes"
Jul 24 21:44:08.476234 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:08.476204    2580 generic.go:356] "Generic (PLEG): container finished" podID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerID="27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2" exitCode=0
Jul 24 21:44:08.476586 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:08.476279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" event={"ID":"b817a8c3-95c5-449a-91ba-3e421f8ed327","Type":"ContainerDied","Data":"27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2"}
Jul 24 21:44:09.482181 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:09.482150    2580 generic.go:356] "Generic (PLEG): container finished" podID="85153d39-9664-4ecc-80db-d30444efdba3" containerID="7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283" exitCode=0
Jul 24 21:44:09.482615 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:09.482229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" event={"ID":"85153d39-9664-4ecc-80db-d30444efdba3","Type":"ContainerDied","Data":"7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283"}
Jul 24 21:44:09.484425 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:09.484406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" event={"ID":"b817a8c3-95c5-449a-91ba-3e421f8ed327","Type":"ContainerStarted","Data":"e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33"}
Jul 24 21:44:09.484500 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:09.484454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" event={"ID":"b817a8c3-95c5-449a-91ba-3e421f8ed327","Type":"ContainerStarted","Data":"7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f"}
Jul 24 21:44:09.484646 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:09.484630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:09.484692 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:09.484682    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:09.486151 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:09.486129    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:44:09.529905 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:09.529849    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podStartSLOduration=2.01553531 podStartE2EDuration="5.529828782s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.514293472s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:44:09.528058196 +0000 UTC m=+1723.866899996" watchObservedRunningTime="2026-07-24 21:44:09.529828782 +0000 UTC m=+1723.868670561"
Jul 24 21:44:10.491462 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:10.491417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" event={"ID":"85153d39-9664-4ecc-80db-d30444efdba3","Type":"ContainerStarted","Data":"59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb"}
Jul 24 21:44:10.491896 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:10.491520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" event={"ID":"85153d39-9664-4ecc-80db-d30444efdba3","Type":"ContainerStarted","Data":"2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4"}
Jul 24 21:44:10.491896 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:10.491795    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:44:10.492040 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:10.491954    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:10.492135 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:10.492114    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:10.493281 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:10.493253    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:44:10.512415 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:10.512342    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podStartSLOduration=2.680864656 podStartE2EDuration="6.512326795s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.831462139s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:44:10.510939428 +0000 UTC m=+1724.849781202" watchObservedRunningTime="2026-07-24 21:44:10.512326795 +0000 UTC m=+1724.851168571"
Jul 24 21:44:11.496741 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:11.496705    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:44:15.495850 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:15.495820    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:44:15.496382 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:15.496357    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:44:16.501539 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:16.501513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:44:16.502055 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:16.502029    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:44:25.497150 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:25.497112    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:44:26.502466 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:26.502427    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:44:35.496889 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:35.496852    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:44:36.502049 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:36.502013    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:44:45.496473 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:45.496436    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:44:46.502930 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:46.502887    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:44:55.496977 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:55.496932    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:44:56.502509 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:44:56.502471    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:45:05.496405 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:05.496346    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:45:06.502678 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:06.502636    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:45:15.496980 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:15.496940    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:45:16.502676 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:16.502593    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:45:25.497058 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:25.497026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:45:26.238195 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:26.238165    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:45:26.239138 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:26.239118    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:45:26.502662 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:26.502576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:45:34.472714 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:34.472677    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"]
Jul 24 21:45:34.473093 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:34.472920    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" containerID="cri-o://2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4" gracePeriod=30
Jul 24 21:45:34.473093 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:34.472964    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kube-rbac-proxy" containerID="cri-o://59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb" gracePeriod=30
Jul 24 21:45:34.904547 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:34.904514    2580 generic.go:356] "Generic (PLEG): container finished" podID="85153d39-9664-4ecc-80db-d30444efdba3" containerID="59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb" exitCode=2
Jul 24 21:45:34.904725 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:34.904560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" event={"ID":"85153d39-9664-4ecc-80db-d30444efdba3","Type":"ContainerDied","Data":"59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb"}
Jul 24 21:45:36.497887 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:36.497845    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 24 21:45:36.502194 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:36.502167    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:45:39.121918 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.121887    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:45:39.160424 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.160377    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-cabundle-cert\") pod \"85153d39-9664-4ecc-80db-d30444efdba3\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") "
Jul 24 21:45:39.160586 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.160467    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85153d39-9664-4ecc-80db-d30444efdba3-proxy-tls\") pod \"85153d39-9664-4ecc-80db-d30444efdba3\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") "
Jul 24 21:45:39.160586 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.160488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-kube-rbac-proxy-sar-config\") pod \"85153d39-9664-4ecc-80db-d30444efdba3\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") "
Jul 24 21:45:39.160586 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.160531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85153d39-9664-4ecc-80db-d30444efdba3-kserve-provision-location\") pod \"85153d39-9664-4ecc-80db-d30444efdba3\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") "
Jul 24 21:45:39.160586 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.160552    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnl6m\" (UniqueName: \"kubernetes.io/projected/85153d39-9664-4ecc-80db-d30444efdba3-kube-api-access-wnl6m\") pod \"85153d39-9664-4ecc-80db-d30444efdba3\" (UID: \"85153d39-9664-4ecc-80db-d30444efdba3\") "
Jul 24 21:45:39.160763 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.160729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85153d39-9664-4ecc-80db-d30444efdba3" (UID: "85153d39-9664-4ecc-80db-d30444efdba3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:45:39.160976 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.160945    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85153d39-9664-4ecc-80db-d30444efdba3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85153d39-9664-4ecc-80db-d30444efdba3" (UID: "85153d39-9664-4ecc-80db-d30444efdba3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:45:39.160976 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.160960    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85153d39-9664-4ecc-80db-d30444efdba3" (UID: "85153d39-9664-4ecc-80db-d30444efdba3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:45:39.160976 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.160963    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:45:39.162851 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.162824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85153d39-9664-4ecc-80db-d30444efdba3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85153d39-9664-4ecc-80db-d30444efdba3" (UID: "85153d39-9664-4ecc-80db-d30444efdba3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:45:39.162851 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.162830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85153d39-9664-4ecc-80db-d30444efdba3-kube-api-access-wnl6m" (OuterVolumeSpecName: "kube-api-access-wnl6m") pod "85153d39-9664-4ecc-80db-d30444efdba3" (UID: "85153d39-9664-4ecc-80db-d30444efdba3"). InnerVolumeSpecName "kube-api-access-wnl6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:45:39.261996 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.261967    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85153d39-9664-4ecc-80db-d30444efdba3-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:45:39.261996 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.261990    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85153d39-9664-4ecc-80db-d30444efdba3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:45:39.261996 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.262000    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85153d39-9664-4ecc-80db-d30444efdba3-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:45:39.262226 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.262011    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnl6m\" (UniqueName: \"kubernetes.io/projected/85153d39-9664-4ecc-80db-d30444efdba3-kube-api-access-wnl6m\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:45:39.931934 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.931900    2580 generic.go:356] "Generic (PLEG): container finished" podID="85153d39-9664-4ecc-80db-d30444efdba3" containerID="2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4" exitCode=0
Jul 24 21:45:39.932136 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.931959    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"
Jul 24 21:45:39.932136 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.931993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" event={"ID":"85153d39-9664-4ecc-80db-d30444efdba3","Type":"ContainerDied","Data":"2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4"}
Jul 24 21:45:39.932136 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.932071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l" event={"ID":"85153d39-9664-4ecc-80db-d30444efdba3","Type":"ContainerDied","Data":"e24f2f55c8a779b4c071b03fd65288a6d097c458cbe6b393f212e1ff9d71db7a"}
Jul 24 21:45:39.932136 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.932134    2580 scope.go:117] "RemoveContainer" containerID="59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb"
Jul 24 21:45:39.941531 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.941508    2580 scope.go:117] "RemoveContainer" containerID="2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4"
Jul 24 21:45:39.949455 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.949436    2580 scope.go:117] "RemoveContainer" containerID="7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283"
Jul 24 21:45:39.956994 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.956962    2580 scope.go:117] "RemoveContainer" containerID="59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb"
Jul 24 21:45:39.957247 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:45:39.957229    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb\": container with ID starting with 59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb not found: ID does not exist" containerID="59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb"
Jul 24 21:45:39.957327 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.957258    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb"} err="failed to get container status \"59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb\": rpc error: code = NotFound desc = could not find container \"59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb\": container with ID starting with 59ae84c434b8f721966ef67eaab32ac7683b473f5ec915105b7b570f54d2bedb not found: ID does not exist"
Jul 24 21:45:39.957327 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.957281    2580 scope.go:117] "RemoveContainer" containerID="2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4"
Jul 24 21:45:39.957547 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:45:39.957529    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4\": container with ID starting with 2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4 not found: ID does not exist" containerID="2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4"
Jul 24 21:45:39.957584 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.957553    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4"} err="failed to get container status \"2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4\": rpc error: code = NotFound desc = could not find container \"2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4\": container with ID starting with 2b3ff2064d9b8ed6ac45938da7855c7d49b814e59e91624121937c0339a610d4 not found: ID does not exist"
Jul 24 21:45:39.957584 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.957567    2580 scope.go:117] "RemoveContainer" containerID="7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283"
Jul 24 21:45:39.957763 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:45:39.957746    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283\": container with ID starting with 7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283 not found: ID does not exist" containerID="7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283"
Jul 24 21:45:39.957820 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.957771    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283"} err="failed to get container status \"7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283\": rpc error: code = NotFound desc = could not find container \"7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283\": container with ID starting with 7c497c7746f0c800a39155e1a085139adcfd20196988743338bde35406877283 not found: ID does not exist"
Jul 24 21:45:39.964957 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.964934    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"]
Jul 24 21:45:39.973521 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:39.973501    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-7b45l"]
Jul 24 21:45:40.287888 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:45:40.287858    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85153d39-9664-4ecc-80db-d30444efdba3" path="/var/lib/kubelet/pods/85153d39-9664-4ecc-80db-d30444efdba3/volumes"
Jul 24 21:47:24.735942 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:24.735852    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"]
Jul 24 21:47:24.738490 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:24.736150    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" containerID="cri-o://7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f" gracePeriod=30
Jul 24 21:47:24.738490 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:24.736169    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kube-rbac-proxy" containerID="cri-o://e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33" gracePeriod=30
Jul 24 21:47:25.419962 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:25.419927    2580 generic.go:356] "Generic (PLEG): container finished" podID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerID="e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33" exitCode=2
Jul 24 21:47:25.420125 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:25.419998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" event={"ID":"b817a8c3-95c5-449a-91ba-3e421f8ed327","Type":"ContainerDied","Data":"e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33"}
Jul 24 21:47:25.492675 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:25.492635    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 24 21:47:25.496966 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:25.496944    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 24 21:47:29.292558 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.292535    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:47:29.445832 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.445742    2580 generic.go:356] "Generic (PLEG): container finished" podID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerID="7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f" exitCode=0
Jul 24 21:47:29.445832 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.445807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"
Jul 24 21:47:29.445832 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.445824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" event={"ID":"b817a8c3-95c5-449a-91ba-3e421f8ed327","Type":"ContainerDied","Data":"7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f"}
Jul 24 21:47:29.446102 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.445889    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq" event={"ID":"b817a8c3-95c5-449a-91ba-3e421f8ed327","Type":"ContainerDied","Data":"a5de979c6ef6cc91f2789806fc583c0c44685cbc119fc98a68991e594634da8c"}
Jul 24 21:47:29.446102 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.445916    2580 scope.go:117] "RemoveContainer" containerID="e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33"
Jul 24 21:47:29.452673 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.452652    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b817a8c3-95c5-449a-91ba-3e421f8ed327-proxy-tls\") pod \"b817a8c3-95c5-449a-91ba-3e421f8ed327\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") "
Jul 24 21:47:29.452798 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.452687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-cabundle-cert\") pod \"b817a8c3-95c5-449a-91ba-3e421f8ed327\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") "
Jul 24 21:47:29.452798 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.452714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hs44\" (UniqueName: \"kubernetes.io/projected/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-api-access-2hs44\") pod \"b817a8c3-95c5-449a-91ba-3e421f8ed327\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") "
Jul 24 21:47:29.452798 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.452756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-rbac-proxy-sar-config\") pod \"b817a8c3-95c5-449a-91ba-3e421f8ed327\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") "
Jul 24 21:47:29.452941 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.452831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b817a8c3-95c5-449a-91ba-3e421f8ed327-kserve-provision-location\") pod \"b817a8c3-95c5-449a-91ba-3e421f8ed327\" (UID: \"b817a8c3-95c5-449a-91ba-3e421f8ed327\") "
Jul 24 21:47:29.453154 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.453120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b817a8c3-95c5-449a-91ba-3e421f8ed327" (UID: "b817a8c3-95c5-449a-91ba-3e421f8ed327"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:47:29.453269 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.453207    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b817a8c3-95c5-449a-91ba-3e421f8ed327" (UID: "b817a8c3-95c5-449a-91ba-3e421f8ed327"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:47:29.453269 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.453230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b817a8c3-95c5-449a-91ba-3e421f8ed327-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b817a8c3-95c5-449a-91ba-3e421f8ed327" (UID: "b817a8c3-95c5-449a-91ba-3e421f8ed327"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:47:29.454933 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.454910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b817a8c3-95c5-449a-91ba-3e421f8ed327-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b817a8c3-95c5-449a-91ba-3e421f8ed327" (UID: "b817a8c3-95c5-449a-91ba-3e421f8ed327"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:47:29.455250 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.455234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-api-access-2hs44" (OuterVolumeSpecName: "kube-api-access-2hs44") pod "b817a8c3-95c5-449a-91ba-3e421f8ed327" (UID: "b817a8c3-95c5-449a-91ba-3e421f8ed327"). InnerVolumeSpecName "kube-api-access-2hs44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:47:29.455321 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.455304    2580 scope.go:117] "RemoveContainer" containerID="7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f"
Jul 24 21:47:29.470311 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.470291    2580 scope.go:117] "RemoveContainer" containerID="27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2"
Jul 24 21:47:29.478181 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.478161    2580 scope.go:117] "RemoveContainer" containerID="e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33"
Jul 24 21:47:29.478473 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:47:29.478448    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33\": container with ID starting with e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33 not found: ID does not exist" containerID="e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33"
Jul 24 21:47:29.478541 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.478488    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33"} err="failed to get container status \"e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33\": rpc error: code = NotFound desc = could not find container \"e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33\": container with ID starting with e6ef9bc8e82eb5f0d99ef614a04d4c387047c79aba463b261d16e21eb0398e33 not found: ID does not exist"
Jul 24 21:47:29.478541 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.478511    2580 scope.go:117] "RemoveContainer" containerID="7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f"
Jul 24 21:47:29.478755 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:47:29.478737    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f\": container with ID starting with 7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f not found: ID does not exist" containerID="7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f"
Jul 24 21:47:29.478804 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.478763    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f"} err="failed to get container status \"7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f\": rpc error: code = NotFound desc = could not find container \"7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f\": container with ID starting with 7dff814ce87ecfde581fd480fbe53bc76903ad5a9f82de8f3ce1dfe024294f2f not found: ID does not exist"
Jul 24 21:47:29.478804 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.478788    2580 scope.go:117] "RemoveContainer" containerID="27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2"
Jul 24 21:47:29.479027 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:47:29.479009    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2\": container with ID starting with 27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2 not found: ID does not exist" containerID="27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2"
Jul 24 21:47:29.479071 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.479032    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2"} err="failed to get container status \"27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2\": rpc error: code = NotFound desc = could not find container \"27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2\": container with ID starting with 27aebadff31345fdf7eeed4d5979bedfd333f96bde383c02700bb06d98571af2 not found: ID does not exist"
Jul 24 21:47:29.553951 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.553916    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.553951 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.553946    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b817a8c3-95c5-449a-91ba-3e421f8ed327-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.553951 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.553957    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b817a8c3-95c5-449a-91ba-3e421f8ed327-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.554158 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.553966    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b817a8c3-95c5-449a-91ba-3e421f8ed327-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.554158 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.553975    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hs44\" (UniqueName: \"kubernetes.io/projected/b817a8c3-95c5-449a-91ba-3e421f8ed327-kube-api-access-2hs44\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.769961 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.769921    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"]
Jul 24 21:47:29.777605 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:29.777573    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-9d2sq"]
Jul 24 21:47:30.287407 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:47:30.287359    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" path="/var/lib/kubelet/pods/b817a8c3-95c5-449a-91ba-3e421f8ed327/volumes"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993018    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"]
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993830    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993850    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993870    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993879    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993889    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kube-rbac-proxy"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993897    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kube-rbac-proxy"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993920    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kube-rbac-proxy"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993927    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kube-rbac-proxy"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993954    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="storage-initializer"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993963    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="storage-initializer"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.993993    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="storage-initializer"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.994001    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="storage-initializer"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.994158    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kube-rbac-proxy"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.994176    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b817a8c3-95c5-449a-91ba-3e421f8ed327" containerName="kserve-container"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.994184    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kube-rbac-proxy"
Jul 24 21:48:59.996479 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.994196    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85153d39-9664-4ecc-80db-d30444efdba3" containerName="kserve-container"
Jul 24 21:48:59.998690 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:48:59.998667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.001352 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.001324    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 24 21:49:00.001495 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.001374    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:49:00.002007 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.001984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:49:00.002495 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.002478    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:49:00.002607 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.002477    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:49:00.002668 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.002629    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:49:00.003052 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.003035    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 24 21:49:00.008435 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.008414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"]
Jul 24 21:49:00.162548 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.162512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhkn\" (UniqueName: \"kubernetes.io/projected/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-api-access-shhkn\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.162723 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.162555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.162723 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.162649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-proxy-tls\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.162723 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.162698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.162843 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.162770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.263587 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.263505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhkn\" (UniqueName: \"kubernetes.io/projected/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-api-access-shhkn\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.263587 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.263550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.263801 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.263622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-proxy-tls\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.263801 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.263658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.263801 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.263702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.264108 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.264083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.264281 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.264260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.264328 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.264279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.266205 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.266183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-proxy-tls\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.273320 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.273296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhkn\" (UniqueName: \"kubernetes.io/projected/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-api-access-shhkn\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-l5l4f\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.310539 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.310506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:00.449260 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.449236    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"]
Jul 24 21:49:00.451653 ip-10-0-129-163 kubenswrapper[2580]: W0724 21:49:00.451616    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d12a051_c3b6_4548_a5d0_a4cdeee691b2.slice/crio-27852d74870199f047eb924ba7fd79eb9c153f83724cc9c3af4b55249e7ce836 WatchSource:0}: Error finding container 27852d74870199f047eb924ba7fd79eb9c153f83724cc9c3af4b55249e7ce836: Status 404 returned error can't find the container with id 27852d74870199f047eb924ba7fd79eb9c153f83724cc9c3af4b55249e7ce836
Jul 24 21:49:00.841320 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.841277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" event={"ID":"5d12a051-c3b6-4548-a5d0-a4cdeee691b2","Type":"ContainerStarted","Data":"925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f"}
Jul 24 21:49:00.841541 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:00.841345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" event={"ID":"5d12a051-c3b6-4548-a5d0-a4cdeee691b2","Type":"ContainerStarted","Data":"27852d74870199f047eb924ba7fd79eb9c153f83724cc9c3af4b55249e7ce836"}
Jul 24 21:49:04.860527 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:04.860496    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerID="925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f" exitCode=0
Jul 24 21:49:04.860924 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:04.860577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" event={"ID":"5d12a051-c3b6-4548-a5d0-a4cdeee691b2","Type":"ContainerDied","Data":"925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f"}
Jul 24 21:49:05.866962 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:05.866924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" event={"ID":"5d12a051-c3b6-4548-a5d0-a4cdeee691b2","Type":"ContainerStarted","Data":"70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a"}
Jul 24 21:49:05.867406 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:05.867010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" event={"ID":"5d12a051-c3b6-4548-a5d0-a4cdeee691b2","Type":"ContainerStarted","Data":"e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823"}
Jul 24 21:49:05.867406 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:05.867222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:05.887774 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:05.887726    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podStartSLOduration=3.078769605 podStartE2EDuration="6.887713176s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.808943571s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:49:05.88583645 +0000 UTC m=+2020.224678225" watchObservedRunningTime="2026-07-24 21:49:05.887713176 +0000 UTC m=+2020.226554951"
Jul 24 21:49:06.872453 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:06.872424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:06.873759 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:06.873729    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:49:07.876909 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:07.876870    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:49:12.881184 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:12.881153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:49:12.881813 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:12.881782    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:49:22.882181 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:22.882137    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:49:32.881964 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:32.881922    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:49:42.881878 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:42.881834    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:49:52.882213 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:49:52.882131    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:50:02.882466 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:02.882429    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:50:12.881785 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:12.881737    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:50:22.883019 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:22.882989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:50:26.269903 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:26.269874    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:50:26.272104 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:26.272082    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:50:30.172463 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:30.172431    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"]
Jul 24 21:50:30.172856 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:30.172673    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" containerID="cri-o://e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823" gracePeriod=30
Jul 24 21:50:30.172856 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:30.172752    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kube-rbac-proxy" containerID="cri-o://70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a" gracePeriod=30
Jul 24 21:50:31.248329 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:31.248294    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerID="70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a" exitCode=2
Jul 24 21:50:31.248730 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:31.248373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" event={"ID":"5d12a051-c3b6-4548-a5d0-a4cdeee691b2","Type":"ContainerDied","Data":"70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a"}
Jul 24 21:50:32.877593 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:32.877547    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 24 21:50:32.881899 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:32.881862    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 24 21:50:35.015640 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.015611    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:50:35.108801 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.108705    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-proxy-tls\") pod \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") "
Jul 24 21:50:35.108801 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.108800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shhkn\" (UniqueName: \"kubernetes.io/projected/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-api-access-shhkn\") pod \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") "
Jul 24 21:50:35.109045 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.108822    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kserve-provision-location\") pod \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") "
Jul 24 21:50:35.109045 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.108845    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-rbac-proxy-sar-config\") pod \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") "
Jul 24 21:50:35.109045 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.108875    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-cabundle-cert\") pod \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\" (UID: \"5d12a051-c3b6-4548-a5d0-a4cdeee691b2\") "
Jul 24 21:50:35.109211 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.109153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d12a051-c3b6-4548-a5d0-a4cdeee691b2" (UID: "5d12a051-c3b6-4548-a5d0-a4cdeee691b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:50:35.109262 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.109249    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d12a051-c3b6-4548-a5d0-a4cdeee691b2" (UID: "5d12a051-c3b6-4548-a5d0-a4cdeee691b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:50:35.109329 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.109302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d12a051-c3b6-4548-a5d0-a4cdeee691b2" (UID: "5d12a051-c3b6-4548-a5d0-a4cdeee691b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:50:35.111066 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.111044    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d12a051-c3b6-4548-a5d0-a4cdeee691b2" (UID: "5d12a051-c3b6-4548-a5d0-a4cdeee691b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:50:35.111167 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.111114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-api-access-shhkn" (OuterVolumeSpecName: "kube-api-access-shhkn") pod "5d12a051-c3b6-4548-a5d0-a4cdeee691b2" (UID: "5d12a051-c3b6-4548-a5d0-a4cdeee691b2"). InnerVolumeSpecName "kube-api-access-shhkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:50:35.210342 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.210305    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhkn\" (UniqueName: \"kubernetes.io/projected/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-api-access-shhkn\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:50:35.210342 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.210342    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:50:35.210581 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.210359    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:50:35.210581 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.210379    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:50:35.210581 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.210417    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d12a051-c3b6-4548-a5d0-a4cdeee691b2-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 21:50:35.269263 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.269224    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerID="e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823" exitCode=0
Jul 24 21:50:35.269474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.269327    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"
Jul 24 21:50:35.269474 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.269341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" event={"ID":"5d12a051-c3b6-4548-a5d0-a4cdeee691b2","Type":"ContainerDied","Data":"e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823"}
Jul 24 21:50:35.269592 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.269473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f" event={"ID":"5d12a051-c3b6-4548-a5d0-a4cdeee691b2","Type":"ContainerDied","Data":"27852d74870199f047eb924ba7fd79eb9c153f83724cc9c3af4b55249e7ce836"}
Jul 24 21:50:35.269592 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.269509    2580 scope.go:117] "RemoveContainer" containerID="70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a"
Jul 24 21:50:35.279528 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.279503    2580 scope.go:117] "RemoveContainer" containerID="e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823"
Jul 24 21:50:35.287651 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.287635    2580 scope.go:117] "RemoveContainer" containerID="925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f"
Jul 24 21:50:35.295594 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.295574    2580 scope.go:117] "RemoveContainer" containerID="70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a"
Jul 24 21:50:35.295851 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:50:35.295828    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a\": container with ID starting with 70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a not found: ID does not exist" containerID="70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a"
Jul 24 21:50:35.295901 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.295866    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a"} err="failed to get container status \"70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a\": rpc error: code = NotFound desc = could not find container \"70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a\": container with ID starting with 70b9c3c7e97a5d934b24ecc54b80ecc67633cd58b6d2d66e9fb9cc128d3df83a not found: ID does not exist"
Jul 24 21:50:35.295901 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.295892    2580 scope.go:117] "RemoveContainer" containerID="e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823"
Jul 24 21:50:35.296155 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:50:35.296138    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823\": container with ID starting with e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823 not found: ID does not exist" containerID="e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823"
Jul 24 21:50:35.296216 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.296162    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823"} err="failed to get container status \"e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823\": rpc error: code = NotFound desc = could not find container \"e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823\": container with ID starting with e80ec2db47cbf27b33caab10642a97b3ad77b5964777fe02fbb6e1a0df4d2823 not found: ID does not exist"
Jul 24 21:50:35.296216 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.296177    2580 scope.go:117] "RemoveContainer" containerID="925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f"
Jul 24 21:50:35.296454 ip-10-0-129-163 kubenswrapper[2580]: E0724 21:50:35.296433    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f\": container with ID starting with 925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f not found: ID does not exist" containerID="925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f"
Jul 24 21:50:35.296546 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.296457    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f"} err="failed to get container status \"925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f\": rpc error: code = NotFound desc = could not find container \"925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f\": container with ID starting with 925d814e7e9733bc6e5718ecb6fc0b6b8bc5a757f650a068a98ed26b7f316d0f not found: ID does not exist"
Jul 24 21:50:35.300165 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.300141    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"]
Jul 24 21:50:35.305819 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:35.305797    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-l5l4f"]
Jul 24 21:50:36.286988 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:50:36.286953    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" path="/var/lib/kubelet/pods/5d12a051-c3b6-4548-a5d0-a4cdeee691b2/volumes"
Jul 24 21:55:26.299965 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:55:26.299939    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 21:55:26.309740 ip-10-0-129-163 kubenswrapper[2580]: I0724 21:55:26.309718    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:00:26.329611 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:00:26.329536    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:00:26.341955 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:00:26.341926    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:01:27.815083 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.815042    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"]
Jul 24 22:01:27.815564 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.815409    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kube-rbac-proxy"
Jul 24 22:01:27.815564 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.815420    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kube-rbac-proxy"
Jul 24 22:01:27.815564 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.815434    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="storage-initializer"
Jul 24 22:01:27.815564 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.815439    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="storage-initializer"
Jul 24 22:01:27.815564 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.815447    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container"
Jul 24 22:01:27.815564 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.815452    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container"
Jul 24 22:01:27.815564 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.815510    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kserve-container"
Jul 24 22:01:27.815564 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.815521    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d12a051-c3b6-4548-a5d0-a4cdeee691b2" containerName="kube-rbac-proxy"
Jul 24 22:01:27.818644 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.818626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.821110 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.821089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 24 22:01:27.821233 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.821112    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:01:27.821233 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.821120    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 24 22:01:27.821318 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.821231    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:01:27.821370 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.821331    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:01:27.821494 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.821471    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:01:27.821616 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.821555    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:01:27.828277 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.828255    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"]
Jul 24 22:01:27.836634 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.836614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktlfj\" (UniqueName: \"kubernetes.io/projected/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-api-access-ktlfj\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.836744 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.836647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.836816 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.836747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.836816 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.836783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.836897 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.836855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.937796 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.937757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.937796 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.937797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.938065 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.937825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.938065 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.937867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktlfj\" (UniqueName: \"kubernetes.io/projected/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-api-access-ktlfj\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.938065 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.937888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.938356 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.938322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.938565 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.938544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.938631 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.938544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.940475 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.940458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:27.945418 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:27.945401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktlfj\" (UniqueName: \"kubernetes.io/projected/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-api-access-ktlfj\") pod \"isvc-paddle-predictor-6d5c9fd4f6-fcrw2\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:28.130463 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:28.130342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:28.265463 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:28.265439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"]
Jul 24 22:01:28.268417 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:01:28.268365    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0110c6_5d44_4ac3_bee4_3867f9582f5a.slice/crio-ad4a4f9cf840761dd7286e72d39b3397a186a2359a6892c81cf11be8acf6896e WatchSource:0}: Error finding container ad4a4f9cf840761dd7286e72d39b3397a186a2359a6892c81cf11be8acf6896e: Status 404 returned error can't find the container with id ad4a4f9cf840761dd7286e72d39b3397a186a2359a6892c81cf11be8acf6896e
Jul 24 22:01:29.077031 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:29.076990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" event={"ID":"cf0110c6-5d44-4ac3-bee4-3867f9582f5a","Type":"ContainerStarted","Data":"d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5"}
Jul 24 22:01:29.077453 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:29.077067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" event={"ID":"cf0110c6-5d44-4ac3-bee4-3867f9582f5a","Type":"ContainerStarted","Data":"ad4a4f9cf840761dd7286e72d39b3397a186a2359a6892c81cf11be8acf6896e"}
Jul 24 22:01:34.099717 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:34.099679    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerID="d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5" exitCode=0
Jul 24 22:01:34.100169 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:34.099752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" event={"ID":"cf0110c6-5d44-4ac3-bee4-3867f9582f5a","Type":"ContainerDied","Data":"d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5"}
Jul 24 22:01:34.100958 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:34.100944    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:01:47.170280 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:47.170245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" event={"ID":"cf0110c6-5d44-4ac3-bee4-3867f9582f5a","Type":"ContainerStarted","Data":"3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a"}
Jul 24 22:01:47.170842 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:47.170311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" event={"ID":"cf0110c6-5d44-4ac3-bee4-3867f9582f5a","Type":"ContainerStarted","Data":"df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45"}
Jul 24 22:01:47.170842 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:47.170646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:47.170842 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:47.170700    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:47.172017 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:47.171990    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 24 22:01:47.190678 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:47.190621    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" podStartSLOduration=2.822101826 podStartE2EDuration="20.190600004s" totalImagesPullingTime="12.558098605s" totalInitContainerRuntime="4.810399573s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:01:47.189867965 +0000 UTC m=+2781.528709751" watchObservedRunningTime="2026-07-24 22:01:47.190600004 +0000 UTC m=+2781.529441781"
Jul 24 22:01:48.175270 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:48.175237    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 24 22:01:53.181837 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:53.181808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:01:53.182497 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:01:53.182472    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 24 22:02:03.182553 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:03.182511    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 24 22:02:13.182959 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:13.182924    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 24 22:02:23.182700 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:23.182662    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 24 22:02:33.183542 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:33.183514    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:02:39.486587 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:39.486545    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"]
Jul 24 22:02:39.487030 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:39.486921    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container" containerID="cri-o://df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45" gracePeriod=30
Jul 24 22:02:39.487030 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:39.486933    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kube-rbac-proxy" containerID="cri-o://3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a" gracePeriod=30
Jul 24 22:02:40.420327 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:40.420294    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerID="3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a" exitCode=2
Jul 24 22:02:40.420327 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:40.420333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" event={"ID":"cf0110c6-5d44-4ac3-bee4-3867f9582f5a","Type":"ContainerDied","Data":"3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a"}
Jul 24 22:02:42.232383 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.232351    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:02:42.321051 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.320972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktlfj\" (UniqueName: \"kubernetes.io/projected/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-api-access-ktlfj\") pod \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") "
Jul 24 22:02:42.321051 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.321005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kserve-provision-location\") pod \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") "
Jul 24 22:02:42.321051 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.321041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-rbac-proxy-sar-config\") pod \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") "
Jul 24 22:02:42.321335 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.321083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-proxy-tls\") pod \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") "
Jul 24 22:02:42.321335 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.321120    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-cabundle-cert\") pod \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\" (UID: \"cf0110c6-5d44-4ac3-bee4-3867f9582f5a\") "
Jul 24 22:02:42.321513 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.321486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf0110c6-5d44-4ac3-bee4-3867f9582f5a" (UID: "cf0110c6-5d44-4ac3-bee4-3867f9582f5a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:02:42.321574 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.321546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf0110c6-5d44-4ac3-bee4-3867f9582f5a" (UID: "cf0110c6-5d44-4ac3-bee4-3867f9582f5a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:02:42.323243 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.323219    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf0110c6-5d44-4ac3-bee4-3867f9582f5a" (UID: "cf0110c6-5d44-4ac3-bee4-3867f9582f5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:02:42.323350 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.323306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-api-access-ktlfj" (OuterVolumeSpecName: "kube-api-access-ktlfj") pod "cf0110c6-5d44-4ac3-bee4-3867f9582f5a" (UID: "cf0110c6-5d44-4ac3-bee4-3867f9582f5a"). InnerVolumeSpecName "kube-api-access-ktlfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:02:42.330864 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.330834    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf0110c6-5d44-4ac3-bee4-3867f9582f5a" (UID: "cf0110c6-5d44-4ac3-bee4-3867f9582f5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:02:42.422052 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.422020    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:02:42.422052 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.422047    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:02:42.422052 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.422057    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktlfj\" (UniqueName: \"kubernetes.io/projected/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-api-access-ktlfj\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:02:42.422253 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.422067    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:02:42.422253 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.422075    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf0110c6-5d44-4ac3-bee4-3867f9582f5a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:02:42.432487 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.432459    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerID="df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45" exitCode=0
Jul 24 22:02:42.432611 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.432510    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"
Jul 24 22:02:42.432611 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.432577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" event={"ID":"cf0110c6-5d44-4ac3-bee4-3867f9582f5a","Type":"ContainerDied","Data":"df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45"}
Jul 24 22:02:42.432687 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.432632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2" event={"ID":"cf0110c6-5d44-4ac3-bee4-3867f9582f5a","Type":"ContainerDied","Data":"ad4a4f9cf840761dd7286e72d39b3397a186a2359a6892c81cf11be8acf6896e"}
Jul 24 22:02:42.432687 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.432659    2580 scope.go:117] "RemoveContainer" containerID="3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a"
Jul 24 22:02:42.441724 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.441706    2580 scope.go:117] "RemoveContainer" containerID="df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45"
Jul 24 22:02:42.449600 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.449583    2580 scope.go:117] "RemoveContainer" containerID="d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5"
Jul 24 22:02:42.454654 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.454633    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"]
Jul 24 22:02:42.457587 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.457571    2580 scope.go:117] "RemoveContainer" containerID="3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a"
Jul 24 22:02:42.457876 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:02:42.457861    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a\": container with ID starting with 3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a not found: ID does not exist" containerID="3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a"
Jul 24 22:02:42.457926 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.457885    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a"} err="failed to get container status \"3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a\": rpc error: code = NotFound desc = could not find container \"3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a\": container with ID starting with 3b5bbcc6aade9d0799452e8109306f378c2713d4db82f175f8177c2386cec94a not found: ID does not exist"
Jul 24 22:02:42.457926 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.457901    2580 scope.go:117] "RemoveContainer" containerID="df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45"
Jul 24 22:02:42.458124 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:02:42.458106    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45\": container with ID starting with df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45 not found: ID does not exist" containerID="df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45"
Jul 24 22:02:42.458171 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.458126    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-fcrw2"]
Jul 24 22:02:42.458171 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.458130    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45"} err="failed to get container status \"df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45\": rpc error: code = NotFound desc = could not find container \"df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45\": container with ID starting with df04f0082bbab2261a7169192c1c124468f4ce142549b0feb35a053510ff6a45 not found: ID does not exist"
Jul 24 22:02:42.458171 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.458143    2580 scope.go:117] "RemoveContainer" containerID="d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5"
Jul 24 22:02:42.458381 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:02:42.458363    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5\": container with ID starting with d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5 not found: ID does not exist" containerID="d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5"
Jul 24 22:02:42.458684 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:42.458663    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5"} err="failed to get container status \"d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5\": rpc error: code = NotFound desc = could not find container \"d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5\": container with ID starting with d1b31d47ac25e2bfff25e47b94bc5c9e8aa32fb8e24d2d9a18171362f95b9be5 not found: ID does not exist"
Jul 24 22:02:44.286892 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:02:44.286862    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" path="/var/lib/kubelet/pods/cf0110c6-5d44-4ac3-bee4-3867f9582f5a/volumes"
Jul 24 22:04:01.431628 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.431598    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"]
Jul 24 22:04:01.432096 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.431943    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container"
Jul 24 22:04:01.432096 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.431953    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container"
Jul 24 22:04:01.432096 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.431963    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kube-rbac-proxy"
Jul 24 22:04:01.432096 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.431968    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kube-rbac-proxy"
Jul 24 22:04:01.432096 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.431983    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="storage-initializer"
Jul 24 22:04:01.432096 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.431989    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="storage-initializer"
Jul 24 22:04:01.432096 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.432050    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kserve-container"
Jul 24 22:04:01.432096 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.432059    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf0110c6-5d44-4ac3-bee4-3867f9582f5a" containerName="kube-rbac-proxy"
Jul 24 22:04:01.434265 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.434250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.436937 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.436916    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 24 22:04:01.437087 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.436933    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:04:01.437087 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.436933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:04:01.437087 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.437012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:04:01.437087 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.436932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:04:01.437926 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.437898    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 24 22:04:01.438048 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.437973    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:04:01.448276 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.448250    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"]
Jul 24 22:04:01.510055 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.510007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.510055 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.510055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.510312 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.510084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba21db9e-666f-4872-b12c-37ebdee8b741-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.510312 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.510139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba21db9e-666f-4872-b12c-37ebdee8b741-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.510312 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.510219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp2jc\" (UniqueName: \"kubernetes.io/projected/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-api-access-sp2jc\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.611305 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.611269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2jc\" (UniqueName: \"kubernetes.io/projected/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-api-access-sp2jc\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.611529 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.611315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.611529 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.611346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.611529 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.611369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba21db9e-666f-4872-b12c-37ebdee8b741-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.611529 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.611455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba21db9e-666f-4872-b12c-37ebdee8b741-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.611911 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.611882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba21db9e-666f-4872-b12c-37ebdee8b741-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.612092 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.612071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.612142 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.612093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.614072 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.614052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba21db9e-666f-4872-b12c-37ebdee8b741-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.620205 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.620181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2jc\" (UniqueName: \"kubernetes.io/projected/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-api-access-sp2jc\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.744871 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.744786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:01.883000 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:01.882965    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"]
Jul 24 22:04:01.885864 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:04:01.885839    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba21db9e_666f_4872_b12c_37ebdee8b741.slice/crio-c523a322728019aa0d24ddd0a05c8353ff100f8a80ccaf294de8ceed40d62893 WatchSource:0}: Error finding container c523a322728019aa0d24ddd0a05c8353ff100f8a80ccaf294de8ceed40d62893: Status 404 returned error can't find the container with id c523a322728019aa0d24ddd0a05c8353ff100f8a80ccaf294de8ceed40d62893
Jul 24 22:04:02.786024 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:02.785986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" event={"ID":"ba21db9e-666f-4872-b12c-37ebdee8b741","Type":"ContainerStarted","Data":"1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf"}
Jul 24 22:04:02.786429 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:02.786050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" event={"ID":"ba21db9e-666f-4872-b12c-37ebdee8b741","Type":"ContainerStarted","Data":"c523a322728019aa0d24ddd0a05c8353ff100f8a80ccaf294de8ceed40d62893"}
Jul 24 22:04:06.807247 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:06.807213    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerID="1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf" exitCode=0
Jul 24 22:04:06.807669 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:06.807287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" event={"ID":"ba21db9e-666f-4872-b12c-37ebdee8b741","Type":"ContainerDied","Data":"1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf"}
Jul 24 22:04:07.814078 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:07.814046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" event={"ID":"ba21db9e-666f-4872-b12c-37ebdee8b741","Type":"ContainerStarted","Data":"ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59"}
Jul 24 22:04:07.814507 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:07.814109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" event={"ID":"ba21db9e-666f-4872-b12c-37ebdee8b741","Type":"ContainerStarted","Data":"482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5"}
Jul 24 22:04:07.814507 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:07.814404    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:07.814618 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:07.814544    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:07.815754 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:07.815731    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 24 22:04:07.837960 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:07.837905    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podStartSLOduration=2.303944477 podStartE2EDuration="6.83788829s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.533943813s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:04:07.836681407 +0000 UTC m=+2922.175523184" watchObservedRunningTime="2026-07-24 22:04:07.83788829 +0000 UTC m=+2922.176730066"
Jul 24 22:04:08.819540 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:08.819498    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 24 22:04:13.824669 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:13.824641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:04:13.825322 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:13.825293    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 24 22:04:23.825628 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:23.825581    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 24 22:04:33.825870 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:33.825827    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 24 22:04:43.826027 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:43.825977    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 24 22:04:53.826619 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:04:53.826532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:05:03.395085 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:03.395055    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"]
Jul 24 22:05:03.395523 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:03.395308    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container" containerID="cri-o://482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5" gracePeriod=30
Jul 24 22:05:03.395523 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:03.395382    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kube-rbac-proxy" containerID="cri-o://ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59" gracePeriod=30
Jul 24 22:05:03.820089 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:03.820049    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 24 22:05:03.825483 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:03.825447    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 24 22:05:04.068986 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:04.068954    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerID="ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59" exitCode=2
Jul 24 22:05:04.069194 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:04.069033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" event={"ID":"ba21db9e-666f-4872-b12c-37ebdee8b741","Type":"ContainerDied","Data":"ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59"}
Jul 24 22:05:06.255133 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.255107    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:05:06.404994 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.404896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba21db9e-666f-4872-b12c-37ebdee8b741-proxy-tls\") pod \"ba21db9e-666f-4872-b12c-37ebdee8b741\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") "
Jul 24 22:05:06.404994 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.404938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-cabundle-cert\") pod \"ba21db9e-666f-4872-b12c-37ebdee8b741\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") "
Jul 24 22:05:06.404994 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.404992    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba21db9e-666f-4872-b12c-37ebdee8b741-kserve-provision-location\") pod \"ba21db9e-666f-4872-b12c-37ebdee8b741\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") "
Jul 24 22:05:06.405286 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.405019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-rbac-proxy-sar-config\") pod \"ba21db9e-666f-4872-b12c-37ebdee8b741\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") "
Jul 24 22:05:06.405286 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.405118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp2jc\" (UniqueName: \"kubernetes.io/projected/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-api-access-sp2jc\") pod \"ba21db9e-666f-4872-b12c-37ebdee8b741\" (UID: \"ba21db9e-666f-4872-b12c-37ebdee8b741\") "
Jul 24 22:05:06.405383 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.405368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba21db9e-666f-4872-b12c-37ebdee8b741" (UID: "ba21db9e-666f-4872-b12c-37ebdee8b741"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:05:06.405511 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.405476    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba21db9e-666f-4872-b12c-37ebdee8b741" (UID: "ba21db9e-666f-4872-b12c-37ebdee8b741"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:05:06.407263 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.407231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba21db9e-666f-4872-b12c-37ebdee8b741-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba21db9e-666f-4872-b12c-37ebdee8b741" (UID: "ba21db9e-666f-4872-b12c-37ebdee8b741"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:05:06.407351 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.407275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-api-access-sp2jc" (OuterVolumeSpecName: "kube-api-access-sp2jc") pod "ba21db9e-666f-4872-b12c-37ebdee8b741" (UID: "ba21db9e-666f-4872-b12c-37ebdee8b741"). InnerVolumeSpecName "kube-api-access-sp2jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:05:06.414522 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.414500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba21db9e-666f-4872-b12c-37ebdee8b741-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba21db9e-666f-4872-b12c-37ebdee8b741" (UID: "ba21db9e-666f-4872-b12c-37ebdee8b741"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:05:06.506284 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.506244    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp2jc\" (UniqueName: \"kubernetes.io/projected/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-api-access-sp2jc\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:05:06.506284 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.506276    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba21db9e-666f-4872-b12c-37ebdee8b741-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:05:06.506284 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.506287    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:05:06.506284 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.506295    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba21db9e-666f-4872-b12c-37ebdee8b741-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:05:06.506588 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:06.506304    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba21db9e-666f-4872-b12c-37ebdee8b741-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:05:07.084089 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.084057    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerID="482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5" exitCode=0
Jul 24 22:05:07.084273 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.084118    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"
Jul 24 22:05:07.084273 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.084130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" event={"ID":"ba21db9e-666f-4872-b12c-37ebdee8b741","Type":"ContainerDied","Data":"482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5"}
Jul 24 22:05:07.084273 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.084180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2" event={"ID":"ba21db9e-666f-4872-b12c-37ebdee8b741","Type":"ContainerDied","Data":"c523a322728019aa0d24ddd0a05c8353ff100f8a80ccaf294de8ceed40d62893"}
Jul 24 22:05:07.084273 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.084208    2580 scope.go:117] "RemoveContainer" containerID="ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59"
Jul 24 22:05:07.093549 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.093526    2580 scope.go:117] "RemoveContainer" containerID="482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5"
Jul 24 22:05:07.101948 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.101926    2580 scope.go:117] "RemoveContainer" containerID="1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf"
Jul 24 22:05:07.105727 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.105702    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"]
Jul 24 22:05:07.110081 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.110056    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-jljp2"]
Jul 24 22:05:07.110780 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.110764    2580 scope.go:117] "RemoveContainer" containerID="ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59"
Jul 24 22:05:07.111113 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:05:07.111098    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59\": container with ID starting with ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59 not found: ID does not exist" containerID="ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59"
Jul 24 22:05:07.111168 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.111122    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59"} err="failed to get container status \"ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59\": rpc error: code = NotFound desc = could not find container \"ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59\": container with ID starting with ff9305fbe66477d8a9eb7157cd11b69a11f5c070574d4f3b3f9f0ae9197a7e59 not found: ID does not exist"
Jul 24 22:05:07.111168 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.111146    2580 scope.go:117] "RemoveContainer" containerID="482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5"
Jul 24 22:05:07.111431 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:05:07.111413    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5\": container with ID starting with 482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5 not found: ID does not exist" containerID="482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5"
Jul 24 22:05:07.111493 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.111436    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5"} err="failed to get container status \"482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5\": rpc error: code = NotFound desc = could not find container \"482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5\": container with ID starting with 482f5a746a73793465fa6504e703c9bd8920141b61598870bb75191af7ee5ca5 not found: ID does not exist"
Jul 24 22:05:07.111493 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.111452    2580 scope.go:117] "RemoveContainer" containerID="1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf"
Jul 24 22:05:07.111696 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:05:07.111681    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf\": container with ID starting with 1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf not found: ID does not exist" containerID="1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf"
Jul 24 22:05:07.111752 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:07.111698    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf"} err="failed to get container status \"1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf\": rpc error: code = NotFound desc = could not find container \"1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf\": container with ID starting with 1e51c14a0113644bdbe0624455748d1405da0866a21dd70662acb5f0fc539cbf not found: ID does not exist"
Jul 24 22:05:08.287408 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:08.287372    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" path="/var/lib/kubelet/pods/ba21db9e-666f-4872-b12c-37ebdee8b741/volumes"
Jul 24 22:05:26.360169 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:26.360141    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:05:26.372407 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:05:26.372366    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:10:26.389403 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:10:26.389357    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:10:26.403591 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:10:26.403564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:15:26.422526 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:15:26.422493    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:15:26.435148 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:15:26.435125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:17:07.013240 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.013204    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"]
Jul 24 22:17:07.013667 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.013618    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kube-rbac-proxy"
Jul 24 22:17:07.013667 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.013630    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kube-rbac-proxy"
Jul 24 22:17:07.013667 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.013653    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="storage-initializer"
Jul 24 22:17:07.013667 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.013661    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="storage-initializer"
Jul 24 22:17:07.013803 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.013676    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container"
Jul 24 22:17:07.013803 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.013683    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container"
Jul 24 22:17:07.013803 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.013748    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kube-rbac-proxy"
Jul 24 22:17:07.013803 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.013758    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba21db9e-666f-4872-b12c-37ebdee8b741" containerName="kserve-container"
Jul 24 22:17:07.016982 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.016963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.019219 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.019194    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:17:07.020109 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.020090    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:17:07.020209 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.020195    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 24 22:17:07.020277 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.020194    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:17:07.020277 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.020242    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:17:07.020406 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.020225    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 24 22:17:07.020406 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.020194    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:17:07.027166 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.027145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"]
Jul 24 22:17:07.064618 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.064578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.064793 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.064627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz55b\" (UniqueName: \"kubernetes.io/projected/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-api-access-gz55b\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.064793 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.064661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.064793 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.064691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.064793 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.064781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.165472 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.165421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.165664 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.165493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.165664 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.165519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz55b\" (UniqueName: \"kubernetes.io/projected/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-api-access-gz55b\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.165664 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.165538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.165664 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.165564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.166042 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.166021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.166271 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.166248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.166334 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.166260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.168019 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.168002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.174592 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.174566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz55b\" (UniqueName: \"kubernetes.io/projected/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-api-access-gz55b\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.327377 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.327346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:07.461958 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:07.461878    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"]
Jul 24 22:17:07.464311 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:17:07.464284    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39571b5c_b9bf_4662_b845_afa5a7fa4b5c.slice/crio-7699651234bbd510db9637f4b9c293d7303aa3fca553e7f3c4dc1eff531eccc7 WatchSource:0}: Error finding container 7699651234bbd510db9637f4b9c293d7303aa3fca553e7f3c4dc1eff531eccc7: Status 404 returned error can't find the container with id 7699651234bbd510db9637f4b9c293d7303aa3fca553e7f3c4dc1eff531eccc7
Jul 24 22:17:08.194955 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:08.194913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" event={"ID":"39571b5c-b9bf-4662-b845-afa5a7fa4b5c","Type":"ContainerStarted","Data":"edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9"}
Jul 24 22:17:08.195340 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:08.194982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" event={"ID":"39571b5c-b9bf-4662-b845-afa5a7fa4b5c","Type":"ContainerStarted","Data":"7699651234bbd510db9637f4b9c293d7303aa3fca553e7f3c4dc1eff531eccc7"}
Jul 24 22:17:12.214602 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:12.214569    2580 generic.go:356] "Generic (PLEG): container finished" podID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerID="edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9" exitCode=0
Jul 24 22:17:12.215022 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:12.214613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" event={"ID":"39571b5c-b9bf-4662-b845-afa5a7fa4b5c","Type":"ContainerDied","Data":"edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9"}
Jul 24 22:17:12.215886 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:12.215871    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:17:33.344885 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:33.344801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" event={"ID":"39571b5c-b9bf-4662-b845-afa5a7fa4b5c","Type":"ContainerStarted","Data":"f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551"}
Jul 24 22:17:33.344885 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:33.344872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" event={"ID":"39571b5c-b9bf-4662-b845-afa5a7fa4b5c","Type":"ContainerStarted","Data":"644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7"}
Jul 24 22:17:33.345346 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:33.345092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:33.367916 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:33.367866    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podStartSLOduration=2.814817435 podStartE2EDuration="27.367851767s" totalImagesPullingTime="20.8300968s" totalInitContainerRuntime="3.722937532s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:17:33.365900391 +0000 UTC m=+3727.704742168" watchObservedRunningTime="2026-07-24 22:17:33.367851767 +0000 UTC m=+3727.706693542"
Jul 24 22:17:34.349122 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:34.349089    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:17:40.357591 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:17:40.357561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:18:10.358500 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:18:10.358460    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 24 22:18:20.358627 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:18:20.358542    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 24 22:18:30.358875 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:18:30.358828    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 24 22:18:40.358684 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:18:40.358634    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 24 22:18:51.398619 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:18:51.398570    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-sklearn-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 22:19:00.360321 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:00.360279    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:19:07.215153 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:07.215113    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"]
Jul 24 22:19:07.215556 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:07.215366    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container" containerID="cri-o://644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7" gracePeriod=30
Jul 24 22:19:07.215556 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:07.215420    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kube-rbac-proxy" containerID="cri-o://f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551" gracePeriod=30
Jul 24 22:19:07.765343 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:07.765302    2580 generic.go:356] "Generic (PLEG): container finished" podID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerID="f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551" exitCode=2
Jul 24 22:19:07.765555 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:07.765381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" event={"ID":"39571b5c-b9bf-4662-b845-afa5a7fa4b5c","Type":"ContainerDied","Data":"f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551"}
Jul 24 22:19:10.353460 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:10.353420    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Jul 24 22:19:10.358811 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:10.358785    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 24 22:19:12.065071 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.065049    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:19:12.189260 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189172    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-proxy-tls\") pod \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") "
Jul 24 22:19:12.189260 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189219    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kserve-provision-location\") pod \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") "
Jul 24 22:19:12.189522 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189276    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-cabundle-cert\") pod \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") "
Jul 24 22:19:12.189522 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189324    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-rbac-proxy-sar-config\") pod \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") "
Jul 24 22:19:12.189522 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz55b\" (UniqueName: \"kubernetes.io/projected/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-api-access-gz55b\") pod \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\" (UID: \"39571b5c-b9bf-4662-b845-afa5a7fa4b5c\") "
Jul 24 22:19:12.189768 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39571b5c-b9bf-4662-b845-afa5a7fa4b5c" (UID: "39571b5c-b9bf-4662-b845-afa5a7fa4b5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:19:12.189858 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39571b5c-b9bf-4662-b845-afa5a7fa4b5c" (UID: "39571b5c-b9bf-4662-b845-afa5a7fa4b5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:19:12.189858 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189853    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:19:12.189974 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189870    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:19:12.189974 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.189917    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39571b5c-b9bf-4662-b845-afa5a7fa4b5c" (UID: "39571b5c-b9bf-4662-b845-afa5a7fa4b5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:19:12.191510 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.191487    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39571b5c-b9bf-4662-b845-afa5a7fa4b5c" (UID: "39571b5c-b9bf-4662-b845-afa5a7fa4b5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:19:12.191750 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.191733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-api-access-gz55b" (OuterVolumeSpecName: "kube-api-access-gz55b") pod "39571b5c-b9bf-4662-b845-afa5a7fa4b5c" (UID: "39571b5c-b9bf-4662-b845-afa5a7fa4b5c"). InnerVolumeSpecName "kube-api-access-gz55b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:19:12.290691 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.290662    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz55b\" (UniqueName: \"kubernetes.io/projected/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kube-api-access-gz55b\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:19:12.290691 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.290690    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:19:12.290894 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.290700    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39571b5c-b9bf-4662-b845-afa5a7fa4b5c-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:19:12.789477 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.789436    2580 generic.go:356] "Generic (PLEG): container finished" podID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerID="644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7" exitCode=0
Jul 24 22:19:12.789656 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.789506    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"
Jul 24 22:19:12.789656 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.789521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" event={"ID":"39571b5c-b9bf-4662-b845-afa5a7fa4b5c","Type":"ContainerDied","Data":"644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7"}
Jul 24 22:19:12.789656 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.789592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg" event={"ID":"39571b5c-b9bf-4662-b845-afa5a7fa4b5c","Type":"ContainerDied","Data":"7699651234bbd510db9637f4b9c293d7303aa3fca553e7f3c4dc1eff531eccc7"}
Jul 24 22:19:12.789656 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.789626    2580 scope.go:117] "RemoveContainer" containerID="f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551"
Jul 24 22:19:12.798108 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.798090    2580 scope.go:117] "RemoveContainer" containerID="644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7"
Jul 24 22:19:12.806278 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.806251    2580 scope.go:117] "RemoveContainer" containerID="edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9"
Jul 24 22:19:12.808745 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.808719    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"]
Jul 24 22:19:12.814147 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.814125    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dkjdg"]
Jul 24 22:19:12.815193 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.815173    2580 scope.go:117] "RemoveContainer" containerID="f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551"
Jul 24 22:19:12.815499 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:19:12.815480    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551\": container with ID starting with f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551 not found: ID does not exist" containerID="f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551"
Jul 24 22:19:12.815571 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.815508    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551"} err="failed to get container status \"f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551\": rpc error: code = NotFound desc = could not find container \"f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551\": container with ID starting with f2ad6d23326d43f1ff95e5d18682196f4af70e8910f09c72cbf8b6ca4b666551 not found: ID does not exist"
Jul 24 22:19:12.815571 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.815526    2580 scope.go:117] "RemoveContainer" containerID="644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7"
Jul 24 22:19:12.815777 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:19:12.815760    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7\": container with ID starting with 644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7 not found: ID does not exist" containerID="644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7"
Jul 24 22:19:12.815821 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.815784    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7"} err="failed to get container status \"644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7\": rpc error: code = NotFound desc = could not find container \"644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7\": container with ID starting with 644d5681784c698769c8f4638a3e6299eb2e7f4f909258120f8cd45c00d491a7 not found: ID does not exist"
Jul 24 22:19:12.815821 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.815802    2580 scope.go:117] "RemoveContainer" containerID="edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9"
Jul 24 22:19:12.816029 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:19:12.816013    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9\": container with ID starting with edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9 not found: ID does not exist" containerID="edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9"
Jul 24 22:19:12.816072 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:12.816036    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9"} err="failed to get container status \"edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9\": rpc error: code = NotFound desc = could not find container \"edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9\": container with ID starting with edf03605a2f01c1031988a38051488d729aac23b1332e654353048192f515dc9 not found: ID does not exist"
Jul 24 22:19:14.286735 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:19:14.286708    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" path="/var/lib/kubelet/pods/39571b5c-b9bf-4662-b845-afa5a7fa4b5c/volumes"
Jul 24 22:20:26.451987 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:20:26.451962    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:20:26.477793 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:20:26.477765    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:25:10.671575 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.671539    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"]
Jul 24 22:25:10.673965 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.671883    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container"
Jul 24 22:25:10.673965 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.671894    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container"
Jul 24 22:25:10.673965 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.671909    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="storage-initializer"
Jul 24 22:25:10.673965 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.671915    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="storage-initializer"
Jul 24 22:25:10.673965 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.671925    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kube-rbac-proxy"
Jul 24 22:25:10.673965 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.671931    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kube-rbac-proxy"
Jul 24 22:25:10.673965 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.671985    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kube-rbac-proxy"
Jul 24 22:25:10.673965 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.671996    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="39571b5c-b9bf-4662-b845-afa5a7fa4b5c" containerName="kserve-container"
Jul 24 22:25:10.675036 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.675022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.677665 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.677637    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 24 22:25:10.677665 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.677637    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:25:10.677847 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.677713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 24 22:25:10.677847 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.677722    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:25:10.677847 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.677780    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:25:10.678077 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.678061    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:25:10.678715 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.678699    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:25:10.684925 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.684906    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"]
Jul 24 22:25:10.840882 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.840845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.840882 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.840891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28657b91-89dc-4cd4-a5d6-80ded64eca77-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.841099 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.840937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.841099 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.840968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cskzb\" (UniqueName: \"kubernetes.io/projected/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-api-access-cskzb\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.841099 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.840994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28657b91-89dc-4cd4-a5d6-80ded64eca77-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.941755 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.941673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.941755 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.941723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28657b91-89dc-4cd4-a5d6-80ded64eca77-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.941945 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.941759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.941945 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.941780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cskzb\" (UniqueName: \"kubernetes.io/projected/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-api-access-cskzb\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.941945 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.941807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28657b91-89dc-4cd4-a5d6-80ded64eca77-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.942302 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.942275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28657b91-89dc-4cd4-a5d6-80ded64eca77-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.942461 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.942441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.942529 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.942460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.944383 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.944364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28657b91-89dc-4cd4-a5d6-80ded64eca77-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.950057 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.950034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cskzb\" (UniqueName: \"kubernetes.io/projected/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-api-access-cskzb\") pod \"isvc-sklearn-runtime-predictor-cf66df4c5-64kkx\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:10.986139 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:10.986086    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:11.121584 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:11.121558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"]
Jul 24 22:25:11.123688 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:25:11.123665    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28657b91_89dc_4cd4_a5d6_80ded64eca77.slice/crio-e9220b95f8f654fab8905f50d7bb59de7ce1d4fc6f45a296b7e0fc7012b9e3e7 WatchSource:0}: Error finding container e9220b95f8f654fab8905f50d7bb59de7ce1d4fc6f45a296b7e0fc7012b9e3e7: Status 404 returned error can't find the container with id e9220b95f8f654fab8905f50d7bb59de7ce1d4fc6f45a296b7e0fc7012b9e3e7
Jul 24 22:25:11.334030 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:11.333995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" event={"ID":"28657b91-89dc-4cd4-a5d6-80ded64eca77","Type":"ContainerStarted","Data":"b92d57aa265e59ee039123a951df501dfb8fdadbd0b893aeb3e8fa318b02d3e4"}
Jul 24 22:25:11.334213 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:11.334052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" event={"ID":"28657b91-89dc-4cd4-a5d6-80ded64eca77","Type":"ContainerStarted","Data":"e9220b95f8f654fab8905f50d7bb59de7ce1d4fc6f45a296b7e0fc7012b9e3e7"}
Jul 24 22:25:17.360319 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:17.360287    2580 generic.go:356] "Generic (PLEG): container finished" podID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerID="b92d57aa265e59ee039123a951df501dfb8fdadbd0b893aeb3e8fa318b02d3e4" exitCode=0
Jul 24 22:25:17.360708 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:17.360359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" event={"ID":"28657b91-89dc-4cd4-a5d6-80ded64eca77","Type":"ContainerDied","Data":"b92d57aa265e59ee039123a951df501dfb8fdadbd0b893aeb3e8fa318b02d3e4"}
Jul 24 22:25:18.367404 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:18.367350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" event={"ID":"28657b91-89dc-4cd4-a5d6-80ded64eca77","Type":"ContainerStarted","Data":"be209ede3d4add2cd33b4f9cc890807eeb41e27a7e5c3c1f27d6ac95d2e9b3fb"}
Jul 24 22:25:18.367831 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:18.367433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" event={"ID":"28657b91-89dc-4cd4-a5d6-80ded64eca77","Type":"ContainerStarted","Data":"6d689fd0808a99bedba2bb23083ecfc47327a588942e570c8eab2b7a829049a0"}
Jul 24 22:25:18.367831 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:18.367708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:18.367955 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:18.367838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:18.369211 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:18.369188    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 24 22:25:19.372547 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:19.372507    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 24 22:25:24.377464 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:24.377434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:24.377923 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:24.377900    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 24 22:25:24.398220 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:24.398163    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" podStartSLOduration=8.707370226 podStartE2EDuration="14.398146703s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.690776477s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:25:18.394249711 +0000 UTC m=+4192.733091511" watchObservedRunningTime="2026-07-24 22:25:24.398146703 +0000 UTC m=+4198.736988478"
Jul 24 22:25:26.484673 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:26.484642    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:25:26.510597 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:26.510569    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:25:34.378562 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:34.378531    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:42.491983 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:42.491950    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-cf66df4c5-64kkx_28657b91-89dc-4cd4-a5d6-80ded64eca77/kserve-container/0.log"
Jul 24 22:25:42.644263 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:42.644229    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"]
Jul 24 22:25:42.644530 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:42.644504    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kserve-container" containerID="cri-o://6d689fd0808a99bedba2bb23083ecfc47327a588942e570c8eab2b7a829049a0" gracePeriod=30
Jul 24 22:25:42.644625 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:42.644537    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kube-rbac-proxy" containerID="cri-o://be209ede3d4add2cd33b4f9cc890807eeb41e27a7e5c3c1f27d6ac95d2e9b3fb" gracePeriod=30
Jul 24 22:25:43.492282 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.492253    2580 generic.go:356] "Generic (PLEG): container finished" podID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerID="be209ede3d4add2cd33b4f9cc890807eeb41e27a7e5c3c1f27d6ac95d2e9b3fb" exitCode=2
Jul 24 22:25:43.492282 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.492276    2580 generic.go:356] "Generic (PLEG): container finished" podID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerID="6d689fd0808a99bedba2bb23083ecfc47327a588942e570c8eab2b7a829049a0" exitCode=0
Jul 24 22:25:43.492648 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.492310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" event={"ID":"28657b91-89dc-4cd4-a5d6-80ded64eca77","Type":"ContainerDied","Data":"be209ede3d4add2cd33b4f9cc890807eeb41e27a7e5c3c1f27d6ac95d2e9b3fb"}
Jul 24 22:25:43.492648 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.492355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" event={"ID":"28657b91-89dc-4cd4-a5d6-80ded64eca77","Type":"ContainerDied","Data":"6d689fd0808a99bedba2bb23083ecfc47327a588942e570c8eab2b7a829049a0"}
Jul 24 22:25:43.492648 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.492374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx" event={"ID":"28657b91-89dc-4cd4-a5d6-80ded64eca77","Type":"ContainerDied","Data":"e9220b95f8f654fab8905f50d7bb59de7ce1d4fc6f45a296b7e0fc7012b9e3e7"}
Jul 24 22:25:43.492648 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.492418    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9220b95f8f654fab8905f50d7bb59de7ce1d4fc6f45a296b7e0fc7012b9e3e7"
Jul 24 22:25:43.501621 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.501600    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:43.626468 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.626325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28657b91-89dc-4cd4-a5d6-80ded64eca77-kserve-provision-location\") pod \"28657b91-89dc-4cd4-a5d6-80ded64eca77\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") "
Jul 24 22:25:43.626468 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.626427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28657b91-89dc-4cd4-a5d6-80ded64eca77-proxy-tls\") pod \"28657b91-89dc-4cd4-a5d6-80ded64eca77\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") "
Jul 24 22:25:43.626701 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.626478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cskzb\" (UniqueName: \"kubernetes.io/projected/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-api-access-cskzb\") pod \"28657b91-89dc-4cd4-a5d6-80ded64eca77\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") "
Jul 24 22:25:43.626701 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.626535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-cabundle-cert\") pod \"28657b91-89dc-4cd4-a5d6-80ded64eca77\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") "
Jul 24 22:25:43.626701 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.626558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-rbac-proxy-sar-config\") pod \"28657b91-89dc-4cd4-a5d6-80ded64eca77\" (UID: \"28657b91-89dc-4cd4-a5d6-80ded64eca77\") "
Jul 24 22:25:43.626915 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.626883    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28657b91-89dc-4cd4-a5d6-80ded64eca77" (UID: "28657b91-89dc-4cd4-a5d6-80ded64eca77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:25:43.627016 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.626936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28657b91-89dc-4cd4-a5d6-80ded64eca77" (UID: "28657b91-89dc-4cd4-a5d6-80ded64eca77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:25:43.628861 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.628843    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-api-access-cskzb" (OuterVolumeSpecName: "kube-api-access-cskzb") pod "28657b91-89dc-4cd4-a5d6-80ded64eca77" (UID: "28657b91-89dc-4cd4-a5d6-80ded64eca77"). InnerVolumeSpecName "kube-api-access-cskzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:25:43.628934 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.628860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28657b91-89dc-4cd4-a5d6-80ded64eca77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28657b91-89dc-4cd4-a5d6-80ded64eca77" (UID: "28657b91-89dc-4cd4-a5d6-80ded64eca77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:25:43.636770 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.636744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28657b91-89dc-4cd4-a5d6-80ded64eca77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28657b91-89dc-4cd4-a5d6-80ded64eca77" (UID: "28657b91-89dc-4cd4-a5d6-80ded64eca77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:25:43.727677 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.727625    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28657b91-89dc-4cd4-a5d6-80ded64eca77-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:25:43.727677 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.727672    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cskzb\" (UniqueName: \"kubernetes.io/projected/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-api-access-cskzb\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:25:43.727677 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.727686    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:25:43.727928 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.727699    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28657b91-89dc-4cd4-a5d6-80ded64eca77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:25:43.727928 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:43.727712    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28657b91-89dc-4cd4-a5d6-80ded64eca77-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:25:44.496679 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:44.496646    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"
Jul 24 22:25:44.514473 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:44.514436    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"]
Jul 24 22:25:44.518399 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:44.518366    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-cf66df4c5-64kkx"]
Jul 24 22:25:46.287372 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:25:46.287302    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" path="/var/lib/kubelet/pods/28657b91-89dc-4cd4-a5d6-80ded64eca77/volumes"
Jul 24 22:30:03.736147 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.736104    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"]
Jul 24 22:30:03.736742 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.736487    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kube-rbac-proxy"
Jul 24 22:30:03.736742 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.736499    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kube-rbac-proxy"
Jul 24 22:30:03.736742 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.736519    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kserve-container"
Jul 24 22:30:03.736742 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.736524    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kserve-container"
Jul 24 22:30:03.736742 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.736537    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="storage-initializer"
Jul 24 22:30:03.736742 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.736542    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="storage-initializer"
Jul 24 22:30:03.736742 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.736594    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kserve-container"
Jul 24 22:30:03.736742 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.736604    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28657b91-89dc-4cd4-a5d6-80ded64eca77" containerName="kube-rbac-proxy"
Jul 24 22:30:03.739781 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.739765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.742145 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.742119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 24 22:30:03.742277 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.742179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:30:03.742336 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.742284    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 24 22:30:03.742495 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.742471    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:30:03.742623 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.742496    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:30:03.742623 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.742577    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:30:03.743146 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.743120    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:30:03.750357 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.750332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"]
Jul 24 22:30:03.837002 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.836969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.837002 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.837005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50eeb3b0-74d6-4908-8c35-6ab237255241-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.837235 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.837034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eeb3b0-74d6-4908-8c35-6ab237255241-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.837235 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.837061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.837235 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.837138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2s2v\" (UniqueName: \"kubernetes.io/projected/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-api-access-q2s2v\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.937991 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.937942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.937991 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.937998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50eeb3b0-74d6-4908-8c35-6ab237255241-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.938241 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.938042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eeb3b0-74d6-4908-8c35-6ab237255241-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.938241 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.938072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.938241 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.938128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2s2v\" (UniqueName: \"kubernetes.io/projected/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-api-access-q2s2v\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.938554 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.938533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50eeb3b0-74d6-4908-8c35-6ab237255241-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.938834 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.938810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.938885 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.938810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.940684 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.940662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eeb3b0-74d6-4908-8c35-6ab237255241-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:03.947022 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:03.947004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2s2v\" (UniqueName: \"kubernetes.io/projected/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-api-access-q2s2v\") pod \"isvc-tensorflow-predictor-547dfc5669-hnppd\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:04.051960 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:04.051921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:04.189206 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:04.189139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"]
Jul 24 22:30:04.191448 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:30:04.191421    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eeb3b0_74d6_4908_8c35_6ab237255241.slice/crio-ce44500d94dbf00eb651109268af17634b2d3b2235998b8b8c762079503f84fa WatchSource:0}: Error finding container ce44500d94dbf00eb651109268af17634b2d3b2235998b8b8c762079503f84fa: Status 404 returned error can't find the container with id ce44500d94dbf00eb651109268af17634b2d3b2235998b8b8c762079503f84fa
Jul 24 22:30:04.645026 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:04.644990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" event={"ID":"50eeb3b0-74d6-4908-8c35-6ab237255241","Type":"ContainerStarted","Data":"a8d5c3fb706a4b1eeb80202c8be57375d48277da9c1e91a201791d54a2fdf935"}
Jul 24 22:30:04.645194 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:04.645063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" event={"ID":"50eeb3b0-74d6-4908-8c35-6ab237255241","Type":"ContainerStarted","Data":"ce44500d94dbf00eb651109268af17634b2d3b2235998b8b8c762079503f84fa"}
Jul 24 22:30:09.670581 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:09.670544    2580 generic.go:356] "Generic (PLEG): container finished" podID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerID="a8d5c3fb706a4b1eeb80202c8be57375d48277da9c1e91a201791d54a2fdf935" exitCode=0
Jul 24 22:30:09.671068 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:09.670588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" event={"ID":"50eeb3b0-74d6-4908-8c35-6ab237255241","Type":"ContainerDied","Data":"a8d5c3fb706a4b1eeb80202c8be57375d48277da9c1e91a201791d54a2fdf935"}
Jul 24 22:30:09.671865 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:09.671846    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:30:13.695314 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:13.695221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" event={"ID":"50eeb3b0-74d6-4908-8c35-6ab237255241","Type":"ContainerStarted","Data":"416ebcc7bdb0c923e4020f94169da14525ee74dcdac2c511c95408653bc8888d"}
Jul 24 22:30:13.695314 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:13.695290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" event={"ID":"50eeb3b0-74d6-4908-8c35-6ab237255241","Type":"ContainerStarted","Data":"561c8c6becb3a19286b068197599e3074e31e9a156f24cb759428832b755cb1c"}
Jul 24 22:30:13.695772 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:13.695621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:13.695772 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:13.695753    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:13.697012 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:13.696986    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 24 22:30:13.718072 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:13.717724    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podStartSLOduration=2.507409419 podStartE2EDuration="10.717702537s" totalImagesPullingTime="3.722437042s" totalInitContainerRuntime="4.487856076s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:30:13.7163164 +0000 UTC m=+4488.055158176" watchObservedRunningTime="2026-07-24 22:30:13.717702537 +0000 UTC m=+4488.056544314"
Jul 24 22:30:14.699852 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:14.699812    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 24 22:30:19.704660 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:19.704587    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:19.705098 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:19.705075    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 24 22:30:26.514382 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:26.514356    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:30:26.545343 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:26.545311    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:30:29.705508 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:29.705475    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:30:44.856905 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:44.856872    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"]
Jul 24 22:30:44.857489 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:44.857141    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kserve-container" containerID="cri-o://561c8c6becb3a19286b068197599e3074e31e9a156f24cb759428832b755cb1c" gracePeriod=30
Jul 24 22:30:44.857489 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:44.857165    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy" containerID="cri-o://416ebcc7bdb0c923e4020f94169da14525ee74dcdac2c511c95408653bc8888d" gracePeriod=30
Jul 24 22:30:45.851830 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:45.851795    2580 generic.go:356] "Generic (PLEG): container finished" podID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerID="416ebcc7bdb0c923e4020f94169da14525ee74dcdac2c511c95408653bc8888d" exitCode=2
Jul 24 22:30:45.852009 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:45.851848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" event={"ID":"50eeb3b0-74d6-4908-8c35-6ab237255241","Type":"ContainerDied","Data":"416ebcc7bdb0c923e4020f94169da14525ee74dcdac2c511c95408653bc8888d"}
Jul 24 22:30:49.700447 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:49.700378    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 24 22:30:54.700634 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:54.700592    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 24 22:30:59.700121 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:59.700067    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 24 22:30:59.700620 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:30:59.700214    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:31:04.700381 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:04.700335    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 24 22:31:09.700918 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:09.700872    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 24 22:31:14.700383 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:14.700341    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 24 22:31:14.992187 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:14.992099    2580 generic.go:356] "Generic (PLEG): container finished" podID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerID="561c8c6becb3a19286b068197599e3074e31e9a156f24cb759428832b755cb1c" exitCode=137
Jul 24 22:31:14.992187 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:14.992174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" event={"ID":"50eeb3b0-74d6-4908-8c35-6ab237255241","Type":"ContainerDied","Data":"561c8c6becb3a19286b068197599e3074e31e9a156f24cb759428832b755cb1c"}
Jul 24 22:31:15.506665 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.506641    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:31:15.552700 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.552667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-rbac-proxy-sar-config\") pod \"50eeb3b0-74d6-4908-8c35-6ab237255241\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") "
Jul 24 22:31:15.552872 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.552720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2s2v\" (UniqueName: \"kubernetes.io/projected/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-api-access-q2s2v\") pod \"50eeb3b0-74d6-4908-8c35-6ab237255241\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") "
Jul 24 22:31:15.552872 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.552749    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50eeb3b0-74d6-4908-8c35-6ab237255241-kserve-provision-location\") pod \"50eeb3b0-74d6-4908-8c35-6ab237255241\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") "
Jul 24 22:31:15.552872 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.552806    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eeb3b0-74d6-4908-8c35-6ab237255241-proxy-tls\") pod \"50eeb3b0-74d6-4908-8c35-6ab237255241\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") "
Jul 24 22:31:15.552872 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.552826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-cabundle-cert\") pod \"50eeb3b0-74d6-4908-8c35-6ab237255241\" (UID: \"50eeb3b0-74d6-4908-8c35-6ab237255241\") "
Jul 24 22:31:15.553172 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.553145    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50eeb3b0-74d6-4908-8c35-6ab237255241" (UID: "50eeb3b0-74d6-4908-8c35-6ab237255241"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:31:15.553263 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.553241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50eeb3b0-74d6-4908-8c35-6ab237255241" (UID: "50eeb3b0-74d6-4908-8c35-6ab237255241"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:31:15.555132 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.555111    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-api-access-q2s2v" (OuterVolumeSpecName: "kube-api-access-q2s2v") pod "50eeb3b0-74d6-4908-8c35-6ab237255241" (UID: "50eeb3b0-74d6-4908-8c35-6ab237255241"). InnerVolumeSpecName "kube-api-access-q2s2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:31:15.555240 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.555149    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50eeb3b0-74d6-4908-8c35-6ab237255241-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50eeb3b0-74d6-4908-8c35-6ab237255241" (UID: "50eeb3b0-74d6-4908-8c35-6ab237255241"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:31:15.563572 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.563548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eeb3b0-74d6-4908-8c35-6ab237255241-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50eeb3b0-74d6-4908-8c35-6ab237255241" (UID: "50eeb3b0-74d6-4908-8c35-6ab237255241"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:31:15.654380 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.654300    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eeb3b0-74d6-4908-8c35-6ab237255241-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:31:15.654380 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.654325    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:31:15.654380 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.654335    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:31:15.654380 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.654344    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2s2v\" (UniqueName: \"kubernetes.io/projected/50eeb3b0-74d6-4908-8c35-6ab237255241-kube-api-access-q2s2v\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:31:15.654380 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.654353    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50eeb3b0-74d6-4908-8c35-6ab237255241-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:31:15.999258 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.999167    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"
Jul 24 22:31:15.999258 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.999179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd" event={"ID":"50eeb3b0-74d6-4908-8c35-6ab237255241","Type":"ContainerDied","Data":"ce44500d94dbf00eb651109268af17634b2d3b2235998b8b8c762079503f84fa"}
Jul 24 22:31:15.999258 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:15.999244    2580 scope.go:117] "RemoveContainer" containerID="416ebcc7bdb0c923e4020f94169da14525ee74dcdac2c511c95408653bc8888d"
Jul 24 22:31:16.008332 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:16.008314    2580 scope.go:117] "RemoveContainer" containerID="561c8c6becb3a19286b068197599e3074e31e9a156f24cb759428832b755cb1c"
Jul 24 22:31:16.016436 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:16.016419    2580 scope.go:117] "RemoveContainer" containerID="a8d5c3fb706a4b1eeb80202c8be57375d48277da9c1e91a201791d54a2fdf935"
Jul 24 22:31:16.021968 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:16.021944    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"]
Jul 24 22:31:16.025687 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:16.025668    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-hnppd"]
Jul 24 22:31:16.286942 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:16.286908    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" path="/var/lib/kubelet/pods/50eeb3b0-74d6-4908-8c35-6ab237255241/volumes"
Jul 24 22:31:26.676713 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:26.676633    2580 scope.go:117] "RemoveContainer" containerID="b92d57aa265e59ee039123a951df501dfb8fdadbd0b893aeb3e8fa318b02d3e4"
Jul 24 22:31:26.685297 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:26.685275    2580 scope.go:117] "RemoveContainer" containerID="be209ede3d4add2cd33b4f9cc890807eeb41e27a7e5c3c1f27d6ac95d2e9b3fb"
Jul 24 22:31:26.695088 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:26.695066    2580 scope.go:117] "RemoveContainer" containerID="6d689fd0808a99bedba2bb23083ecfc47327a588942e570c8eab2b7a829049a0"
Jul 24 22:31:35.504231 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.504198    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"]
Jul 24 22:31:35.504601 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.504567    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kserve-container"
Jul 24 22:31:35.504601 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.504578    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kserve-container"
Jul 24 22:31:35.504601 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.504597    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy"
Jul 24 22:31:35.504601 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.504603    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy"
Jul 24 22:31:35.504733 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.504611    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="storage-initializer"
Jul 24 22:31:35.504733 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.504617    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="storage-initializer"
Jul 24 22:31:35.504733 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.504671    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kube-rbac-proxy"
Jul 24 22:31:35.504733 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.504680    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50eeb3b0-74d6-4908-8c35-6ab237255241" containerName="kserve-container"
Jul 24 22:31:35.509257 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.509239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.511947 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.511923    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:31:35.512062 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.511992    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:31:35.512062 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.512003    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:31:35.512062 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.511922    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:31:35.512895 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.512865    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:31:35.512994 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.512901    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 24 22:31:35.512994 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.512910    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 24 22:31:35.519407 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.519371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"]
Jul 24 22:31:35.634443 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.634413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.634620 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.634457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.634620 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.634478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsr85\" (UniqueName: \"kubernetes.io/projected/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-api-access-lsr85\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.634620 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.634557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.634620 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.634585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.735951 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.735914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.736146 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.735960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.736146 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.735981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsr85\" (UniqueName: \"kubernetes.io/projected/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-api-access-lsr85\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.736146 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.736007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.736146 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.736023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.736429 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.736378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.736914 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.736893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.736975 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.736921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.738692 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.738673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.744169 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.744147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsr85\" (UniqueName: \"kubernetes.io/projected/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-api-access-lsr85\") pod \"isvc-triton-predictor-5d7db58676-cztrf\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.820412 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.820351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:31:35.952009 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:35.951941    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"]
Jul 24 22:31:35.954449 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:31:35.954421    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbda88a4_8fc7_4eb8_863f_21eb1d60b7b6.slice/crio-54cf656a563122a35c2767d37fcaa64493ba028a85e1be24417b8aa3ae702729 WatchSource:0}: Error finding container 54cf656a563122a35c2767d37fcaa64493ba028a85e1be24417b8aa3ae702729: Status 404 returned error can't find the container with id 54cf656a563122a35c2767d37fcaa64493ba028a85e1be24417b8aa3ae702729
Jul 24 22:31:36.090848 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:36.090755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" event={"ID":"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6","Type":"ContainerStarted","Data":"6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d"}
Jul 24 22:31:36.090848 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:36.090828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" event={"ID":"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6","Type":"ContainerStarted","Data":"54cf656a563122a35c2767d37fcaa64493ba028a85e1be24417b8aa3ae702729"}
Jul 24 22:31:40.110724 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:40.110690    2580 generic.go:356] "Generic (PLEG): container finished" podID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerID="6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d" exitCode=0
Jul 24 22:31:40.111119 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:31:40.110770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" event={"ID":"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6","Type":"ContainerDied","Data":"6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d"}
Jul 24 22:33:36.775671 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:36.775630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" event={"ID":"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6","Type":"ContainerStarted","Data":"37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4"}
Jul 24 22:33:36.776081 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:36.775688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" event={"ID":"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6","Type":"ContainerStarted","Data":"3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210"}
Jul 24 22:33:36.776081 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:36.775741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:33:36.804934 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:36.804881    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" podStartSLOduration=2.539459886 podStartE2EDuration="2m1.80486669s" totalImagesPullingTime="1m55.567929394s" totalInitContainerRuntime="3.69747741s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:33:36.803599607 +0000 UTC m=+4691.142441396" watchObservedRunningTime="2026-07-24 22:33:36.80486669 +0000 UTC m=+4691.143708466"
Jul 24 22:33:37.780255 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:37.780220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:33:37.781599 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:37.781570    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 24 22:33:38.784562 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:38.784525    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 24 22:33:43.789317 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:43.789289    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:33:43.790027 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:43.790009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:33:47.897850 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:47.897819    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"]
Jul 24 22:33:47.898306 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:47.898184    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kserve-container" containerID="cri-o://3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210" gracePeriod=30
Jul 24 22:33:47.898306 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:47.898196    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kube-rbac-proxy" containerID="cri-o://37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4" gracePeriod=30
Jul 24 22:33:48.784724 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:48.784681    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 24 22:33:48.832166 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:48.832134    2580 generic.go:356] "Generic (PLEG): container finished" podID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerID="37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4" exitCode=2
Jul 24 22:33:48.832329 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:48.832174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" event={"ID":"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6","Type":"ContainerDied","Data":"37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4"}
Jul 24 22:33:51.104545 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.104511    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:33:51.253659 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.253626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-proxy-tls\") pod \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") "
Jul 24 22:33:51.253659 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.253661    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsr85\" (UniqueName: \"kubernetes.io/projected/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-api-access-lsr85\") pod \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") "
Jul 24 22:33:51.253863 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.253709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kserve-provision-location\") pod \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") "
Jul 24 22:33:51.253863 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.253737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-rbac-proxy-sar-config\") pod \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") "
Jul 24 22:33:51.253863 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.253756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-cabundle-cert\") pod \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\" (UID: \"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6\") "
Jul 24 22:33:51.254109 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.254084    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" (UID: "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:33:51.254170 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.254151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" (UID: "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:33:51.254238 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.254217    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" (UID: "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:33:51.255985 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.255964    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" (UID: "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:33:51.256084 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.256056    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-api-access-lsr85" (OuterVolumeSpecName: "kube-api-access-lsr85") pod "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" (UID: "fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6"). InnerVolumeSpecName "kube-api-access-lsr85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:33:51.354891 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.354812    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:33:51.354891 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.354841    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsr85\" (UniqueName: \"kubernetes.io/projected/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-api-access-lsr85\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:33:51.354891 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.354851    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:33:51.354891 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.354860    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:33:51.354891 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.354869    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:33:51.847230 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.847187    2580 generic.go:356] "Generic (PLEG): container finished" podID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerID="3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210" exitCode=0
Jul 24 22:33:51.847469 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.847240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" event={"ID":"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6","Type":"ContainerDied","Data":"3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210"}
Jul 24 22:33:51.847469 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.847254    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"
Jul 24 22:33:51.847469 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.847287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf" event={"ID":"fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6","Type":"ContainerDied","Data":"54cf656a563122a35c2767d37fcaa64493ba028a85e1be24417b8aa3ae702729"}
Jul 24 22:33:51.847469 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.847313    2580 scope.go:117] "RemoveContainer" containerID="37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4"
Jul 24 22:33:51.858610 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.858584    2580 scope.go:117] "RemoveContainer" containerID="3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210"
Jul 24 22:33:51.867468 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.867448    2580 scope.go:117] "RemoveContainer" containerID="6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d"
Jul 24 22:33:51.868838 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.868820    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"]
Jul 24 22:33:51.872458 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.872435    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-cztrf"]
Jul 24 22:33:51.884503 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.884467    2580 scope.go:117] "RemoveContainer" containerID="37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4"
Jul 24 22:33:51.884816 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:33:51.884797    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4\": container with ID starting with 37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4 not found: ID does not exist" containerID="37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4"
Jul 24 22:33:51.884894 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.884830    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4"} err="failed to get container status \"37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4\": rpc error: code = NotFound desc = could not find container \"37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4\": container with ID starting with 37f1d1f2e26d48d1d5002140fd698924a8dbbd5571b0f30a2d8c1ca2b4d4a8b4 not found: ID does not exist"
Jul 24 22:33:51.884894 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.884855    2580 scope.go:117] "RemoveContainer" containerID="3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210"
Jul 24 22:33:51.885145 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:33:51.885128    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210\": container with ID starting with 3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210 not found: ID does not exist" containerID="3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210"
Jul 24 22:33:51.885183 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.885156    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210"} err="failed to get container status \"3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210\": rpc error: code = NotFound desc = could not find container \"3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210\": container with ID starting with 3acef377a89ef915326bac1d586f69aedf0f0aad4edd2a518de682f956a2e210 not found: ID does not exist"
Jul 24 22:33:51.885183 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.885174    2580 scope.go:117] "RemoveContainer" containerID="6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d"
Jul 24 22:33:51.885417 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:33:51.885375    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d\": container with ID starting with 6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d not found: ID does not exist" containerID="6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d"
Jul 24 22:33:51.885483 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:51.885428    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d"} err="failed to get container status \"6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d\": rpc error: code = NotFound desc = could not find container \"6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d\": container with ID starting with 6ed9d6fd3831907a2633f4dc758fc976c9b33eb2b7a7f204af9dbc537a23094d not found: ID does not exist"
Jul 24 22:33:52.286715 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:33:52.286685    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" path="/var/lib/kubelet/pods/fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6/volumes"
Jul 24 22:35:26.549714 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:35:26.549690    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:35:26.574901 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:35:26.574876    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:40:26.578281 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:40:26.578254    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:40:26.608238 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:40:26.608210    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:42:15.013425 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.013376    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"]
Jul 24 22:42:15.013896 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.013778    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="storage-initializer"
Jul 24 22:42:15.013896 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.013789    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="storage-initializer"
Jul 24 22:42:15.013896 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.013801    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kube-rbac-proxy"
Jul 24 22:42:15.013896 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.013806    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kube-rbac-proxy"
Jul 24 22:42:15.013896 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.013819    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kserve-container"
Jul 24 22:42:15.013896 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.013824    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kserve-container"
Jul 24 22:42:15.013896 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.013884    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kube-rbac-proxy"
Jul 24 22:42:15.013896 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.013891    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbda88a4-8fc7-4eb8-863f-21eb1d60b7b6" containerName="kserve-container"
Jul 24 22:42:15.016081 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.016067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.018971 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.018942    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 24 22:42:15.018971 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.018949    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 24 22:42:15.019161 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.019037    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:42:15.019161 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.019070    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 24 22:42:15.019161 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.019119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:42:15.019347 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.019330    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:42:15.019415 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.019336    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:42:15.028317 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.028293    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"]
Jul 24 22:42:15.113577 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.113540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.113768 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.113595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.113768 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.113657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9gt\" (UniqueName: \"kubernetes.io/projected/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-api-access-bq9gt\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.113768 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.113677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.113768 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.113730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fc06b2-f8d5-4549-8037-3a6f7677c890-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.214190 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.214151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.214372 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.214207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.214372 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.214243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9gt\" (UniqueName: \"kubernetes.io/projected/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-api-access-bq9gt\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.214372 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.214261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.214372 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.214285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fc06b2-f8d5-4549-8037-3a6f7677c890-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.214706 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.214683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.214909 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.214882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.215011 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.214924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.216910 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.216893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fc06b2-f8d5-4549-8037-3a6f7677c890-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.224714 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.224690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9gt\" (UniqueName: \"kubernetes.io/projected/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-api-access-bq9gt\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.327989 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.327951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:15.461739 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:15.461715    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"]
Jul 24 22:42:15.464085 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:42:15.464056    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fc06b2_f8d5_4549_8037_3a6f7677c890.slice/crio-80673a017c2805ae47dbffffc494b401da212151493fc32e37736d69d7a43e1d WatchSource:0}: Error finding container 80673a017c2805ae47dbffffc494b401da212151493fc32e37736d69d7a43e1d: Status 404 returned error can't find the container with id 80673a017c2805ae47dbffffc494b401da212151493fc32e37736d69d7a43e1d
Jul 24 22:42:16.032082 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:16.032035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" event={"ID":"e8fc06b2-f8d5-4549-8037-3a6f7677c890","Type":"ContainerStarted","Data":"d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb"}
Jul 24 22:42:16.032538 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:16.032108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" event={"ID":"e8fc06b2-f8d5-4549-8037-3a6f7677c890","Type":"ContainerStarted","Data":"80673a017c2805ae47dbffffc494b401da212151493fc32e37736d69d7a43e1d"}
Jul 24 22:42:17.038089 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:17.038056    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerID="d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb" exitCode=0
Jul 24 22:42:17.038511 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:17.038139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" event={"ID":"e8fc06b2-f8d5-4549-8037-3a6f7677c890","Type":"ContainerDied","Data":"d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb"}
Jul 24 22:42:18.044914 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:18.044880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" event={"ID":"e8fc06b2-f8d5-4549-8037-3a6f7677c890","Type":"ContainerStarted","Data":"10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6"}
Jul 24 22:42:18.045329 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:18.045080    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:18.045329 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:18.045122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" event={"ID":"e8fc06b2-f8d5-4549-8037-3a6f7677c890","Type":"ContainerStarted","Data":"df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c"}
Jul 24 22:42:18.068328 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:18.068277    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podStartSLOduration=2.779627626 podStartE2EDuration="4.068261285s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.288633659s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:42:18.066938325 +0000 UTC m=+5212.405780115" watchObservedRunningTime="2026-07-24 22:42:18.068261285 +0000 UTC m=+5212.407103060"
Jul 24 22:42:19.050177 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:19.050141    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:19.051546 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:19.051517    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 24 22:42:20.054578 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:20.054530    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 24 22:42:25.060733 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:25.060706    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:42:25.061322 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:25.061298    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 24 22:42:35.061706 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:35.061664    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 24 22:42:45.061835 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:45.061792    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 24 22:42:55.061558 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:42:55.061518    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 24 22:43:05.061979 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:05.061933    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 24 22:43:15.062248 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:15.062205    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 24 22:43:25.061725 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:25.061679    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 24 22:43:35.062554 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:35.062525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:43:45.065317 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:45.065287    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"]
Jul 24 22:43:45.065789 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:45.065572    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container" containerID="cri-o://df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c" gracePeriod=30
Jul 24 22:43:45.065789 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:45.065601    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kube-rbac-proxy" containerID="cri-o://10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6" gracePeriod=30
Jul 24 22:43:45.460650 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:45.460564    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerID="10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6" exitCode=2
Jul 24 22:43:45.460650 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:45.460635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" event={"ID":"e8fc06b2-f8d5-4549-8037-3a6f7677c890","Type":"ContainerDied","Data":"10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6"}
Jul 24 22:43:49.909156 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:49.909128    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:43:50.077451 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.077420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kserve-provision-location\") pod \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") "
Jul 24 22:43:50.077659 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.077468    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-cabundle-cert\") pod \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") "
Jul 24 22:43:50.077659 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.077491    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq9gt\" (UniqueName: \"kubernetes.io/projected/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-api-access-bq9gt\") pod \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") "
Jul 24 22:43:50.077659 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.077555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fc06b2-f8d5-4549-8037-3a6f7677c890-proxy-tls\") pod \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") "
Jul 24 22:43:50.077659 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.077573    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-rbac-proxy-sar-config\") pod \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\" (UID: \"e8fc06b2-f8d5-4549-8037-3a6f7677c890\") "
Jul 24 22:43:50.077883 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.077804    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8fc06b2-f8d5-4549-8037-3a6f7677c890" (UID: "e8fc06b2-f8d5-4549-8037-3a6f7677c890"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:43:50.077965 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.077835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8fc06b2-f8d5-4549-8037-3a6f7677c890" (UID: "e8fc06b2-f8d5-4549-8037-3a6f7677c890"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:43:50.078005 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.077986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8fc06b2-f8d5-4549-8037-3a6f7677c890" (UID: "e8fc06b2-f8d5-4549-8037-3a6f7677c890"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:43:50.079761 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.079733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fc06b2-f8d5-4549-8037-3a6f7677c890-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8fc06b2-f8d5-4549-8037-3a6f7677c890" (UID: "e8fc06b2-f8d5-4549-8037-3a6f7677c890"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:43:50.079761 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.079750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-api-access-bq9gt" (OuterVolumeSpecName: "kube-api-access-bq9gt") pod "e8fc06b2-f8d5-4549-8037-3a6f7677c890" (UID: "e8fc06b2-f8d5-4549-8037-3a6f7677c890"). InnerVolumeSpecName "kube-api-access-bq9gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:43:50.179152 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.179116    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:43:50.179152 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.179145    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-cabundle-cert\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:43:50.179412 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.179155    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9gt\" (UniqueName: \"kubernetes.io/projected/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-api-access-bq9gt\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:43:50.179412 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.179182    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fc06b2-f8d5-4549-8037-3a6f7677c890-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:43:50.179412 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.179204    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8fc06b2-f8d5-4549-8037-3a6f7677c890-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:43:50.486539 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.486444    2580 generic.go:356] "Generic (PLEG): container finished" podID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerID="df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c" exitCode=0
Jul 24 22:43:50.486539 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.486506    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"
Jul 24 22:43:50.486716 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.486536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" event={"ID":"e8fc06b2-f8d5-4549-8037-3a6f7677c890","Type":"ContainerDied","Data":"df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c"}
Jul 24 22:43:50.486716 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.486626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8" event={"ID":"e8fc06b2-f8d5-4549-8037-3a6f7677c890","Type":"ContainerDied","Data":"80673a017c2805ae47dbffffc494b401da212151493fc32e37736d69d7a43e1d"}
Jul 24 22:43:50.486716 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.486654    2580 scope.go:117] "RemoveContainer" containerID="10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6"
Jul 24 22:43:50.495256 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.495239    2580 scope.go:117] "RemoveContainer" containerID="df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c"
Jul 24 22:43:50.502829 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.502809    2580 scope.go:117] "RemoveContainer" containerID="d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb"
Jul 24 22:43:50.505176 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.505157    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"]
Jul 24 22:43:50.510993 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.510968    2580 scope.go:117] "RemoveContainer" containerID="10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6"
Jul 24 22:43:50.511346 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:43:50.511322    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6\": container with ID starting with 10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6 not found: ID does not exist" containerID="10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6"
Jul 24 22:43:50.511443 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.511355    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6"} err="failed to get container status \"10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6\": rpc error: code = NotFound desc = could not find container \"10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6\": container with ID starting with 10a9357a70f7d28fbb10a1538b77fea3d98e22fc3640bfed18d5322c4b5a33f6 not found: ID does not exist"
Jul 24 22:43:50.511443 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.511375    2580 scope.go:117] "RemoveContainer" containerID="df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c"
Jul 24 22:43:50.511684 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:43:50.511664    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c\": container with ID starting with df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c not found: ID does not exist" containerID="df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c"
Jul 24 22:43:50.511767 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.511694    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c"} err="failed to get container status \"df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c\": rpc error: code = NotFound desc = could not find container \"df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c\": container with ID starting with df95bb38fde7e61b17a28d2f1874271ab38043dd7426a6f6609b6974b3efb87c not found: ID does not exist"
Jul 24 22:43:50.511767 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.511720    2580 scope.go:117] "RemoveContainer" containerID="d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb"
Jul 24 22:43:50.511980 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:43:50.511962    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb\": container with ID starting with d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb not found: ID does not exist" containerID="d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb"
Jul 24 22:43:50.512028 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.511984    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb"} err="failed to get container status \"d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb\": rpc error: code = NotFound desc = could not find container \"d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb\": container with ID starting with d87364af67618fac6ecc120742f5ca39eaef4689025974731ee378f73ec86aeb not found: ID does not exist"
Jul 24 22:43:50.512675 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:50.512657    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-6d644d499d-dm7t8"]
Jul 24 22:43:52.286648 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:43:52.286612    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" path="/var/lib/kubelet/pods/e8fc06b2-f8d5-4549-8037-3a6f7677c890/volumes"
Jul 24 22:45:18.359974 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.359894    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"]
Jul 24 22:45:18.360485 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.360242    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="storage-initializer"
Jul 24 22:45:18.360485 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.360252    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="storage-initializer"
Jul 24 22:45:18.360485 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.360260    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container"
Jul 24 22:45:18.360485 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.360265    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container"
Jul 24 22:45:18.360485 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.360287    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kube-rbac-proxy"
Jul 24 22:45:18.360485 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.360293    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kube-rbac-proxy"
Jul 24 22:45:18.360485 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.360342    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kube-rbac-proxy"
Jul 24 22:45:18.360485 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.360351    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fc06b2-f8d5-4549-8037-3a6f7677c890" containerName="kserve-container"
Jul 24 22:45:18.363615 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.363600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.365831 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.365812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 24 22:45:18.366025 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.366010    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:45:18.366081 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.366064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:45:18.366081 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.366069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 24 22:45:18.367018 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.367006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:45:18.367111 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.367069    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 24 22:45:18.372924 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.372901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"]
Jul 24 22:45:18.429673 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.429639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/932c59b3-7e1f-43c6-92d8-22824f30a729-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.429673 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.429676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvf4\" (UniqueName: \"kubernetes.io/projected/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-api-access-5mvf4\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.429890 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.429713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.429890 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.429764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/932c59b3-7e1f-43c6-92d8-22824f30a729-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.531026 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.530994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/932c59b3-7e1f-43c6-92d8-22824f30a729-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.531229 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.531072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/932c59b3-7e1f-43c6-92d8-22824f30a729-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.531229 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.531092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvf4\" (UniqueName: \"kubernetes.io/projected/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-api-access-5mvf4\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.531229 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.531115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.531443 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:45:18.531245    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 24 22:45:18.531443 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:45:18.531326    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932c59b3-7e1f-43c6-92d8-22824f30a729-proxy-tls podName:932c59b3-7e1f-43c6-92d8-22824f30a729 nodeName:}" failed. No retries permitted until 2026-07-24 22:45:19.031302311 +0000 UTC m=+5393.370144086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/932c59b3-7e1f-43c6-92d8-22824f30a729-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj" (UID: "932c59b3-7e1f-43c6-92d8-22824f30a729") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 24 22:45:18.531655 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.531567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/932c59b3-7e1f-43c6-92d8-22824f30a729-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.531861 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.531845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:18.539426 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:18.539379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvf4\" (UniqueName: \"kubernetes.io/projected/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-api-access-5mvf4\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:19.035769 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:19.035735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/932c59b3-7e1f-43c6-92d8-22824f30a729-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:19.038371 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:19.038352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/932c59b3-7e1f-43c6-92d8-22824f30a729-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:19.274832 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:19.274798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:19.410947 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:19.410923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"]
Jul 24 22:45:19.413826 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:45:19.413800    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932c59b3_7e1f_43c6_92d8_22824f30a729.slice/crio-9c2a1aa4e4e60902d18b5ea5758ace804b1eb72940ebc27dba5ae7af969fb567 WatchSource:0}: Error finding container 9c2a1aa4e4e60902d18b5ea5758ace804b1eb72940ebc27dba5ae7af969fb567: Status 404 returned error can't find the container with id 9c2a1aa4e4e60902d18b5ea5758ace804b1eb72940ebc27dba5ae7af969fb567
Jul 24 22:45:19.891907 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:19.891871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj" event={"ID":"932c59b3-7e1f-43c6-92d8-22824f30a729","Type":"ContainerStarted","Data":"d3c0e8de6f32e0444df3424f4abb29d8fb0df649ec7b05ba3d5b6e274400d0c8"}
Jul 24 22:45:19.892156 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:19.891924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj" event={"ID":"932c59b3-7e1f-43c6-92d8-22824f30a729","Type":"ContainerStarted","Data":"9c2a1aa4e4e60902d18b5ea5758ace804b1eb72940ebc27dba5ae7af969fb567"}
Jul 24 22:45:22.907628 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:22.907598    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_932c59b3-7e1f-43c6-92d8-22824f30a729/storage-initializer/0.log"
Jul 24 22:45:22.908091 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:22.907640    2580 generic.go:356] "Generic (PLEG): container finished" podID="932c59b3-7e1f-43c6-92d8-22824f30a729" containerID="d3c0e8de6f32e0444df3424f4abb29d8fb0df649ec7b05ba3d5b6e274400d0c8" exitCode=1
Jul 24 22:45:22.908091 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:22.907678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj" event={"ID":"932c59b3-7e1f-43c6-92d8-22824f30a729","Type":"ContainerDied","Data":"d3c0e8de6f32e0444df3424f4abb29d8fb0df649ec7b05ba3d5b6e274400d0c8"}
Jul 24 22:45:23.913738 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:23.913664    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_932c59b3-7e1f-43c6-92d8-22824f30a729/storage-initializer/0.log"
Jul 24 22:45:23.914127 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:23.913750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj" event={"ID":"932c59b3-7e1f-43c6-92d8-22824f30a729","Type":"ContainerStarted","Data":"68b6048e24d83d83cdfbfa90fe04ee836ac5424639276b51670ea86f4f19fd32"}
Jul 24 22:45:26.594137 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:26.594026    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_932c59b3-7e1f-43c6-92d8-22824f30a729/storage-initializer/0.log"
Jul 24 22:45:26.611592 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:26.611571    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:45:26.621151 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:26.621127    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_932c59b3-7e1f-43c6-92d8-22824f30a729/storage-initializer/0.log"
Jul 24 22:45:26.638167 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:26.638148    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:45:27.935725 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:27.935696    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_932c59b3-7e1f-43c6-92d8-22824f30a729/storage-initializer/1.log"
Jul 24 22:45:27.936129 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:27.936067    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_932c59b3-7e1f-43c6-92d8-22824f30a729/storage-initializer/0.log"
Jul 24 22:45:27.936129 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:27.936098    2580 generic.go:356] "Generic (PLEG): container finished" podID="932c59b3-7e1f-43c6-92d8-22824f30a729" containerID="68b6048e24d83d83cdfbfa90fe04ee836ac5424639276b51670ea86f4f19fd32" exitCode=1
Jul 24 22:45:27.936218 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:27.936176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj" event={"ID":"932c59b3-7e1f-43c6-92d8-22824f30a729","Type":"ContainerDied","Data":"68b6048e24d83d83cdfbfa90fe04ee836ac5424639276b51670ea86f4f19fd32"}
Jul 24 22:45:27.936265 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:27.936242    2580 scope.go:117] "RemoveContainer" containerID="d3c0e8de6f32e0444df3424f4abb29d8fb0df649ec7b05ba3d5b6e274400d0c8"
Jul 24 22:45:27.936647 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:27.936627    2580 scope.go:117] "RemoveContainer" containerID="d3c0e8de6f32e0444df3424f4abb29d8fb0df649ec7b05ba3d5b6e274400d0c8"
Jul 24 22:45:27.947728 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:45:27.947707    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_kserve-ci-e2e-test(932c59b3-7e1f-43c6-92d8-22824f30a729)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj" podUID="932c59b3-7e1f-43c6-92d8-22824f30a729"
Jul 24 22:45:27.947833 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:45:27.947809    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_kserve-ci-e2e-test_932c59b3-7e1f-43c6-92d8-22824f30a729_0 in pod sandbox 9c2a1aa4e4e60902d18b5ea5758ace804b1eb72940ebc27dba5ae7af969fb567 from index: no such id: 'd3c0e8de6f32e0444df3424f4abb29d8fb0df649ec7b05ba3d5b6e274400d0c8'" containerID="d3c0e8de6f32e0444df3424f4abb29d8fb0df649ec7b05ba3d5b6e274400d0c8"
Jul 24 22:45:27.947891 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:27.947843    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c0e8de6f32e0444df3424f4abb29d8fb0df649ec7b05ba3d5b6e274400d0c8"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_kserve-ci-e2e-test_932c59b3-7e1f-43c6-92d8-22824f30a729_0 in pod sandbox 9c2a1aa4e4e60902d18b5ea5758ace804b1eb72940ebc27dba5ae7af969fb567 from index: no such id: 'd3c0e8de6f32e0444df3424f4abb29d8fb0df649ec7b05ba3d5b6e274400d0c8'"
Jul 24 22:45:28.369169 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:28.369137    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"]
Jul 24 22:45:28.941996 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:28.941967    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_932c59b3-7e1f-43c6-92d8-22824f30a729/storage-initializer/1.log"
Jul 24 22:45:29.072618 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.072595    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_932c59b3-7e1f-43c6-92d8-22824f30a729/storage-initializer/1.log"
Jul 24 22:45:29.076091 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.076074    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:29.227583 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.227496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/932c59b3-7e1f-43c6-92d8-22824f30a729-proxy-tls\") pod \"932c59b3-7e1f-43c6-92d8-22824f30a729\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") "
Jul 24 22:45:29.227583 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.227539    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mvf4\" (UniqueName: \"kubernetes.io/projected/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-api-access-5mvf4\") pod \"932c59b3-7e1f-43c6-92d8-22824f30a729\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") "
Jul 24 22:45:29.227583 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.227579    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-rbac-proxy-sar-config\") pod \"932c59b3-7e1f-43c6-92d8-22824f30a729\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") "
Jul 24 22:45:29.227869 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.227652    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/932c59b3-7e1f-43c6-92d8-22824f30a729-kserve-provision-location\") pod \"932c59b3-7e1f-43c6-92d8-22824f30a729\" (UID: \"932c59b3-7e1f-43c6-92d8-22824f30a729\") "
Jul 24 22:45:29.227967 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.227942    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932c59b3-7e1f-43c6-92d8-22824f30a729-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "932c59b3-7e1f-43c6-92d8-22824f30a729" (UID: "932c59b3-7e1f-43c6-92d8-22824f30a729"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:45:29.228022 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.227994    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "932c59b3-7e1f-43c6-92d8-22824f30a729" (UID: "932c59b3-7e1f-43c6-92d8-22824f30a729"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:45:29.229815 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.229792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932c59b3-7e1f-43c6-92d8-22824f30a729-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "932c59b3-7e1f-43c6-92d8-22824f30a729" (UID: "932c59b3-7e1f-43c6-92d8-22824f30a729"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:45:29.229921 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.229812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-api-access-5mvf4" (OuterVolumeSpecName: "kube-api-access-5mvf4") pod "932c59b3-7e1f-43c6-92d8-22824f30a729" (UID: "932c59b3-7e1f-43c6-92d8-22824f30a729"). InnerVolumeSpecName "kube-api-access-5mvf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:45:29.328632 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.328596    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/932c59b3-7e1f-43c6-92d8-22824f30a729-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:45:29.328632 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.328625    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/932c59b3-7e1f-43c6-92d8-22824f30a729-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:45:29.328632 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.328639    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mvf4\" (UniqueName: \"kubernetes.io/projected/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-api-access-5mvf4\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:45:29.328873 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.328652    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/932c59b3-7e1f-43c6-92d8-22824f30a729-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:45:29.947651 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.947625    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj_932c59b3-7e1f-43c6-92d8-22824f30a729/storage-initializer/1.log"
Jul 24 22:45:29.948064 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.947728    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"
Jul 24 22:45:29.948064 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.947734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj" event={"ID":"932c59b3-7e1f-43c6-92d8-22824f30a729","Type":"ContainerDied","Data":"9c2a1aa4e4e60902d18b5ea5758ace804b1eb72940ebc27dba5ae7af969fb567"}
Jul 24 22:45:29.948064 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.947796    2580 scope.go:117] "RemoveContainer" containerID="68b6048e24d83d83cdfbfa90fe04ee836ac5424639276b51670ea86f4f19fd32"
Jul 24 22:45:29.985597 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.985559    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"]
Jul 24 22:45:29.988312 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:29.988289    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b47d47bc4-vjlpj"]
Jul 24 22:45:30.287345 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:45:30.287313    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932c59b3-7e1f-43c6-92d8-22824f30a729" path="/var/lib/kubelet/pods/932c59b3-7e1f-43c6-92d8-22824f30a729/volumes"
Jul 24 22:47:00.622370 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.622328    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"]
Jul 24 22:47:00.622905 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.622874    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932c59b3-7e1f-43c6-92d8-22824f30a729" containerName="storage-initializer"
Jul 24 22:47:00.622905 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.622894    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="932c59b3-7e1f-43c6-92d8-22824f30a729" containerName="storage-initializer"
Jul 24 22:47:00.623031 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.622931    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932c59b3-7e1f-43c6-92d8-22824f30a729" containerName="storage-initializer"
Jul 24 22:47:00.623031 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.622940    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="932c59b3-7e1f-43c6-92d8-22824f30a729" containerName="storage-initializer"
Jul 24 22:47:00.623135 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.623028    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="932c59b3-7e1f-43c6-92d8-22824f30a729" containerName="storage-initializer"
Jul 24 22:47:00.623135 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.623042    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="932c59b3-7e1f-43c6-92d8-22824f30a729" containerName="storage-initializer"
Jul 24 22:47:00.626428 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.626382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.628758 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.628734    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 24 22:47:00.628872 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.628851    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 24 22:47:00.629060 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.629048    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:47:00.629185 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.629170    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:47:00.629687 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.629673    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:47:00.629739 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.629677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 24 22:47:00.635253 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.635226    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"]
Jul 24 22:47:00.717164 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.717123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zklb\" (UniqueName: \"kubernetes.io/projected/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-api-access-2zklb\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.717164 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.717163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9d41004-86b4-44ce-8f05-ac8e0683d139-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.717376 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.717276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9d41004-86b4-44ce-8f05-ac8e0683d139-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.717376 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.717343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.818264 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.818232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9d41004-86b4-44ce-8f05-ac8e0683d139-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.818415 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.818293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.818415 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.818321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zklb\" (UniqueName: \"kubernetes.io/projected/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-api-access-2zklb\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.818415 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.818341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9d41004-86b4-44ce-8f05-ac8e0683d139-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.818415 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:47:00.818381    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 24 22:47:00.818632 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:47:00.818492    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9d41004-86b4-44ce-8f05-ac8e0683d139-proxy-tls podName:e9d41004-86b4-44ce-8f05-ac8e0683d139 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:01.318470319 +0000 UTC m=+5495.657312087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e9d41004-86b4-44ce-8f05-ac8e0683d139-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q" (UID: "e9d41004-86b4-44ce-8f05-ac8e0683d139") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 24 22:47:00.818737 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.818721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9d41004-86b4-44ce-8f05-ac8e0683d139-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.818936 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.818917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:00.826798 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:00.826741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zklb\" (UniqueName: \"kubernetes.io/projected/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-api-access-2zklb\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:01.323190 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:01.323153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9d41004-86b4-44ce-8f05-ac8e0683d139-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:01.325714 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:01.325692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9d41004-86b4-44ce-8f05-ac8e0683d139-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:01.537430 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:01.537370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:01.672782 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:01.672686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"]
Jul 24 22:47:01.675707 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:47:01.675678    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d41004_86b4_44ce_8f05_ac8e0683d139.slice/crio-050b3b73a625fb3c4e90c09f855564265b59bfa90fc8a0fe227c630a4fc56683 WatchSource:0}: Error finding container 050b3b73a625fb3c4e90c09f855564265b59bfa90fc8a0fe227c630a4fc56683: Status 404 returned error can't find the container with id 050b3b73a625fb3c4e90c09f855564265b59bfa90fc8a0fe227c630a4fc56683
Jul 24 22:47:02.376236 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:02.376195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q" event={"ID":"e9d41004-86b4-44ce-8f05-ac8e0683d139","Type":"ContainerStarted","Data":"8651bcd8290145531048c255de41575e4ffc817f43cfa37dfc86c7a37c0c76c7"}
Jul 24 22:47:02.376435 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:02.376275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q" event={"ID":"e9d41004-86b4-44ce-8f05-ac8e0683d139","Type":"ContainerStarted","Data":"050b3b73a625fb3c4e90c09f855564265b59bfa90fc8a0fe227c630a4fc56683"}
Jul 24 22:47:05.390508 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:05.390480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_e9d41004-86b4-44ce-8f05-ac8e0683d139/storage-initializer/0.log"
Jul 24 22:47:05.390909 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:05.390519    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9d41004-86b4-44ce-8f05-ac8e0683d139" containerID="8651bcd8290145531048c255de41575e4ffc817f43cfa37dfc86c7a37c0c76c7" exitCode=1
Jul 24 22:47:05.390909 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:05.390601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q" event={"ID":"e9d41004-86b4-44ce-8f05-ac8e0683d139","Type":"ContainerDied","Data":"8651bcd8290145531048c255de41575e4ffc817f43cfa37dfc86c7a37c0c76c7"}
Jul 24 22:47:06.397240 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:06.397214    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_e9d41004-86b4-44ce-8f05-ac8e0683d139/storage-initializer/0.log"
Jul 24 22:47:06.397707 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:06.397259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q" event={"ID":"e9d41004-86b4-44ce-8f05-ac8e0683d139","Type":"ContainerStarted","Data":"74dd6044d6c9132bde5e05ce75769e7dbcb05609818897229fee76567260dcae"}
Jul 24 22:47:09.412258 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:09.412237    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_e9d41004-86b4-44ce-8f05-ac8e0683d139/storage-initializer/1.log"
Jul 24 22:47:09.412618 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:09.412604    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_e9d41004-86b4-44ce-8f05-ac8e0683d139/storage-initializer/0.log"
Jul 24 22:47:09.412666 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:09.412637    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9d41004-86b4-44ce-8f05-ac8e0683d139" containerID="74dd6044d6c9132bde5e05ce75769e7dbcb05609818897229fee76567260dcae" exitCode=1
Jul 24 22:47:09.412738 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:09.412668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q" event={"ID":"e9d41004-86b4-44ce-8f05-ac8e0683d139","Type":"ContainerDied","Data":"74dd6044d6c9132bde5e05ce75769e7dbcb05609818897229fee76567260dcae"}
Jul 24 22:47:09.412738 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:09.412722    2580 scope.go:117] "RemoveContainer" containerID="8651bcd8290145531048c255de41575e4ffc817f43cfa37dfc86c7a37c0c76c7"
Jul 24 22:47:09.413118 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:09.413104    2580 scope.go:117] "RemoveContainer" containerID="8651bcd8290145531048c255de41575e4ffc817f43cfa37dfc86c7a37c0c76c7"
Jul 24 22:47:09.447573 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:47:09.447546    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_kserve-ci-e2e-test(e9d41004-86b4-44ce-8f05-ac8e0683d139)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q" podUID="e9d41004-86b4-44ce-8f05-ac8e0683d139"
Jul 24 22:47:09.447700 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:47:09.447578    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_kserve-ci-e2e-test_e9d41004-86b4-44ce-8f05-ac8e0683d139_0 in pod sandbox 050b3b73a625fb3c4e90c09f855564265b59bfa90fc8a0fe227c630a4fc56683 from index: no such id: '8651bcd8290145531048c255de41575e4ffc817f43cfa37dfc86c7a37c0c76c7'" containerID="8651bcd8290145531048c255de41575e4ffc817f43cfa37dfc86c7a37c0c76c7"
Jul 24 22:47:09.447700 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:09.447610    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8651bcd8290145531048c255de41575e4ffc817f43cfa37dfc86c7a37c0c76c7"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_kserve-ci-e2e-test_e9d41004-86b4-44ce-8f05-ac8e0683d139_0 in pod sandbox 050b3b73a625fb3c4e90c09f855564265b59bfa90fc8a0fe227c630a4fc56683 from index: no such id: '8651bcd8290145531048c255de41575e4ffc817f43cfa37dfc86c7a37c0c76c7'"
Jul 24 22:47:10.418973 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.418945    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_e9d41004-86b4-44ce-8f05-ac8e0683d139/storage-initializer/1.log"
Jul 24 22:47:10.608528 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.608494    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"]
Jul 24 22:47:10.739185 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.739164    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_e9d41004-86b4-44ce-8f05-ac8e0683d139/storage-initializer/1.log"
Jul 24 22:47:10.742667 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.742651    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:10.906368 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.906335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zklb\" (UniqueName: \"kubernetes.io/projected/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-api-access-2zklb\") pod \"e9d41004-86b4-44ce-8f05-ac8e0683d139\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") "
Jul 24 22:47:10.906581 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.906451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-rbac-proxy-sar-config\") pod \"e9d41004-86b4-44ce-8f05-ac8e0683d139\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") "
Jul 24 22:47:10.906581 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.906491    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9d41004-86b4-44ce-8f05-ac8e0683d139-kserve-provision-location\") pod \"e9d41004-86b4-44ce-8f05-ac8e0683d139\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") "
Jul 24 22:47:10.906581 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.906514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9d41004-86b4-44ce-8f05-ac8e0683d139-proxy-tls\") pod \"e9d41004-86b4-44ce-8f05-ac8e0683d139\" (UID: \"e9d41004-86b4-44ce-8f05-ac8e0683d139\") "
Jul 24 22:47:10.906819 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.906772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d41004-86b4-44ce-8f05-ac8e0683d139-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9d41004-86b4-44ce-8f05-ac8e0683d139" (UID: "e9d41004-86b4-44ce-8f05-ac8e0683d139"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:47:10.906885 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.906858    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9d41004-86b4-44ce-8f05-ac8e0683d139" (UID: "e9d41004-86b4-44ce-8f05-ac8e0683d139"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:47:10.908684 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.908657    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-api-access-2zklb" (OuterVolumeSpecName: "kube-api-access-2zklb") pod "e9d41004-86b4-44ce-8f05-ac8e0683d139" (UID: "e9d41004-86b4-44ce-8f05-ac8e0683d139"). InnerVolumeSpecName "kube-api-access-2zklb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:47:10.908885 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:10.908871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d41004-86b4-44ce-8f05-ac8e0683d139-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9d41004-86b4-44ce-8f05-ac8e0683d139" (UID: "e9d41004-86b4-44ce-8f05-ac8e0683d139"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:47:11.007920 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.007826    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:47:11.007920 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.007865    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9d41004-86b4-44ce-8f05-ac8e0683d139-kserve-provision-location\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:47:11.007920 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.007879    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9d41004-86b4-44ce-8f05-ac8e0683d139-proxy-tls\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:47:11.007920 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.007892    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zklb\" (UniqueName: \"kubernetes.io/projected/e9d41004-86b4-44ce-8f05-ac8e0683d139-kube-api-access-2zklb\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:47:11.424767 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.424742    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q_e9d41004-86b4-44ce-8f05-ac8e0683d139/storage-initializer/1.log"
Jul 24 22:47:11.425251 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.424850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q" event={"ID":"e9d41004-86b4-44ce-8f05-ac8e0683d139","Type":"ContainerDied","Data":"050b3b73a625fb3c4e90c09f855564265b59bfa90fc8a0fe227c630a4fc56683"}
Jul 24 22:47:11.425251 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.424854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"
Jul 24 22:47:11.425251 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.424905    2580 scope.go:117] "RemoveContainer" containerID="74dd6044d6c9132bde5e05ce75769e7dbcb05609818897229fee76567260dcae"
Jul 24 22:47:11.459582 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.459553    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"]
Jul 24 22:47:11.464971 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:11.464944    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6694cc88b7-pj54q"]
Jul 24 22:47:12.287823 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.287790    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d41004-86b4-44ce-8f05-ac8e0683d139" path="/var/lib/kubelet/pods/e9d41004-86b4-44ce-8f05-ac8e0683d139/volumes"
Jul 24 22:47:12.338169 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.338136    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cfjts/must-gather-nvrzc"]
Jul 24 22:47:12.338496 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.338484    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d41004-86b4-44ce-8f05-ac8e0683d139" containerName="storage-initializer"
Jul 24 22:47:12.338551 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.338498    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d41004-86b4-44ce-8f05-ac8e0683d139" containerName="storage-initializer"
Jul 24 22:47:12.338551 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.338515    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d41004-86b4-44ce-8f05-ac8e0683d139" containerName="storage-initializer"
Jul 24 22:47:12.338551 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.338521    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d41004-86b4-44ce-8f05-ac8e0683d139" containerName="storage-initializer"
Jul 24 22:47:12.338645 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.338596    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d41004-86b4-44ce-8f05-ac8e0683d139" containerName="storage-initializer"
Jul 24 22:47:12.338645 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.338605    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d41004-86b4-44ce-8f05-ac8e0683d139" containerName="storage-initializer"
Jul 24 22:47:12.343403 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.343370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:12.345681 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.345660    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfjts\"/\"kube-root-ca.crt\""
Jul 24 22:47:12.345797 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.345681    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cfjts\"/\"openshift-service-ca.crt\""
Jul 24 22:47:12.350594 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.350189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfjts/must-gather-nvrzc"]
Jul 24 22:47:12.420157 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.420122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-must-gather-output\") pod \"must-gather-nvrzc\" (UID: \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\") " pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:12.420325 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.420168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bmmb\" (UniqueName: \"kubernetes.io/projected/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-kube-api-access-2bmmb\") pod \"must-gather-nvrzc\" (UID: \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\") " pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:12.520707 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.520675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-must-gather-output\") pod \"must-gather-nvrzc\" (UID: \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\") " pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:12.520707 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.520719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bmmb\" (UniqueName: \"kubernetes.io/projected/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-kube-api-access-2bmmb\") pod \"must-gather-nvrzc\" (UID: \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\") " pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:12.521151 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.521002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-must-gather-output\") pod \"must-gather-nvrzc\" (UID: \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\") " pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:12.528490 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.528469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bmmb\" (UniqueName: \"kubernetes.io/projected/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-kube-api-access-2bmmb\") pod \"must-gather-nvrzc\" (UID: \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\") " pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:12.664641 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.664539    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:12.794115 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.794090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cfjts/must-gather-nvrzc"]
Jul 24 22:47:12.796223 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:47:12.796196    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cf00eb8_1d65_44cf_83de_e1a5f8b2bff6.slice/crio-1a3dbd25de830642f72d2b70fb0bb2002204677d03611b2dad7f5949c5052a13 WatchSource:0}: Error finding container 1a3dbd25de830642f72d2b70fb0bb2002204677d03611b2dad7f5949c5052a13: Status 404 returned error can't find the container with id 1a3dbd25de830642f72d2b70fb0bb2002204677d03611b2dad7f5949c5052a13
Jul 24 22:47:12.797891 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:12.797876    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:47:13.440457 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:13.440356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfjts/must-gather-nvrzc" event={"ID":"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6","Type":"ContainerStarted","Data":"1a3dbd25de830642f72d2b70fb0bb2002204677d03611b2dad7f5949c5052a13"}
Jul 24 22:47:19.476202 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:19.476080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfjts/must-gather-nvrzc" event={"ID":"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6","Type":"ContainerStarted","Data":"c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949"}
Jul 24 22:47:19.476202 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:19.476124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfjts/must-gather-nvrzc" event={"ID":"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6","Type":"ContainerStarted","Data":"c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353"}
Jul 24 22:47:19.492880 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:19.492829    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cfjts/must-gather-nvrzc" podStartSLOduration=1.461039899 podStartE2EDuration="7.492814752s" totalImagesPullingTime="6.031774853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:19.491347911 +0000 UTC m=+5513.830189687" watchObservedRunningTime="2026-07-24 22:47:19.492814752 +0000 UTC m=+5513.831656571"
Jul 24 22:47:45.609373 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:45.609290    2580 generic.go:356] "Generic (PLEG): container finished" podID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" containerID="c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353" exitCode=0
Jul 24 22:47:45.609373 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:45.609363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cfjts/must-gather-nvrzc" event={"ID":"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6","Type":"ContainerDied","Data":"c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353"}
Jul 24 22:47:45.609830 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:45.609681    2580 scope.go:117] "RemoveContainer" containerID="c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353"
Jul 24 22:47:46.318830 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:46.318795    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cfjts_must-gather-nvrzc_6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6/gather/0.log"
Jul 24 22:47:47.032248 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.032217    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mcx84/must-gather-92kpl"]
Jul 24 22:47:47.035036 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.035020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcx84/must-gather-92kpl"
Jul 24 22:47:47.037344 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.037321    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mcx84\"/\"default-dockercfg-zm8wh\""
Jul 24 22:47:47.038271 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.038251    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mcx84\"/\"kube-root-ca.crt\""
Jul 24 22:47:47.038271 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.038264    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mcx84\"/\"openshift-service-ca.crt\""
Jul 24 22:47:47.043259 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.043240    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcx84/must-gather-92kpl"]
Jul 24 22:47:47.151799 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.151762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c754a44a-81ea-48fd-93d4-7e457960c14b-must-gather-output\") pod \"must-gather-92kpl\" (UID: \"c754a44a-81ea-48fd-93d4-7e457960c14b\") " pod="openshift-must-gather-mcx84/must-gather-92kpl"
Jul 24 22:47:47.151799 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.151795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wnq\" (UniqueName: \"kubernetes.io/projected/c754a44a-81ea-48fd-93d4-7e457960c14b-kube-api-access-x5wnq\") pod \"must-gather-92kpl\" (UID: \"c754a44a-81ea-48fd-93d4-7e457960c14b\") " pod="openshift-must-gather-mcx84/must-gather-92kpl"
Jul 24 22:47:47.252992 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.252961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c754a44a-81ea-48fd-93d4-7e457960c14b-must-gather-output\") pod \"must-gather-92kpl\" (UID: \"c754a44a-81ea-48fd-93d4-7e457960c14b\") " pod="openshift-must-gather-mcx84/must-gather-92kpl"
Jul 24 22:47:47.252992 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.253000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wnq\" (UniqueName: \"kubernetes.io/projected/c754a44a-81ea-48fd-93d4-7e457960c14b-kube-api-access-x5wnq\") pod \"must-gather-92kpl\" (UID: \"c754a44a-81ea-48fd-93d4-7e457960c14b\") " pod="openshift-must-gather-mcx84/must-gather-92kpl"
Jul 24 22:47:47.253349 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.253328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c754a44a-81ea-48fd-93d4-7e457960c14b-must-gather-output\") pod \"must-gather-92kpl\" (UID: \"c754a44a-81ea-48fd-93d4-7e457960c14b\") " pod="openshift-must-gather-mcx84/must-gather-92kpl"
Jul 24 22:47:47.262210 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.262181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wnq\" (UniqueName: \"kubernetes.io/projected/c754a44a-81ea-48fd-93d4-7e457960c14b-kube-api-access-x5wnq\") pod \"must-gather-92kpl\" (UID: \"c754a44a-81ea-48fd-93d4-7e457960c14b\") " pod="openshift-must-gather-mcx84/must-gather-92kpl"
Jul 24 22:47:47.345183 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.345092    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcx84/must-gather-92kpl"
Jul 24 22:47:47.477038 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.477009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcx84/must-gather-92kpl"]
Jul 24 22:47:47.479749 ip-10-0-129-163 kubenswrapper[2580]: W0724 22:47:47.479721    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc754a44a_81ea_48fd_93d4_7e457960c14b.slice/crio-3e253d1f53b8be32486947eb10fb3494f279060c305a1bb421e1cbf1110075c7 WatchSource:0}: Error finding container 3e253d1f53b8be32486947eb10fb3494f279060c305a1bb421e1cbf1110075c7: Status 404 returned error can't find the container with id 3e253d1f53b8be32486947eb10fb3494f279060c305a1bb421e1cbf1110075c7
Jul 24 22:47:47.619788 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:47.619697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcx84/must-gather-92kpl" event={"ID":"c754a44a-81ea-48fd-93d4-7e457960c14b","Type":"ContainerStarted","Data":"3e253d1f53b8be32486947eb10fb3494f279060c305a1bb421e1cbf1110075c7"}
Jul 24 22:47:48.626916 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:48.626877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcx84/must-gather-92kpl" event={"ID":"c754a44a-81ea-48fd-93d4-7e457960c14b","Type":"ContainerStarted","Data":"fa16a23ddbc38043b39240c79ead56ca7c774d5f7c4001ca59c1d71554c5051e"}
Jul 24 22:47:48.627347 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:48.626926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcx84/must-gather-92kpl" event={"ID":"c754a44a-81ea-48fd-93d4-7e457960c14b","Type":"ContainerStarted","Data":"7c732720426788476935e5847d68f6b2b75bdb10c291d2ef0fc871e35cdedb64"}
Jul 24 22:47:48.650749 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:48.650687    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mcx84/must-gather-92kpl" podStartSLOduration=0.929829695 podStartE2EDuration="1.650665941s" totalImagesPullingTime="720.836246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:48.647870559 +0000 UTC m=+5542.986712337" watchObservedRunningTime="2026-07-24 22:47:48.650665941 +0000 UTC m=+5542.989507720"
Jul 24 22:47:49.941549 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:49.941510    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xc9pb_6df404c6-1f92-408c-b16b-8697d4f86fdc/global-pull-secret-syncer/0.log"
Jul 24 22:47:50.064098 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:50.064062    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7v2tr_4bf8bea7-f8c3-4ebf-9e1c-0618e4da92a2/konnectivity-agent/0.log"
Jul 24 22:47:50.170481 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:50.170438    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-163.ec2.internal_a258609098d82a70d5e5404dc8cfce27/haproxy/0.log"
Jul 24 22:47:51.880479 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:51.880437    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-cfjts/must-gather-nvrzc"]
Jul 24 22:47:51.881407 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:51.881357    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-cfjts/must-gather-nvrzc" podUID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" containerName="copy" containerID="cri-o://c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949" gracePeriod=2
Jul 24 22:47:51.886840 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:51.886803    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-cfjts/must-gather-nvrzc"]
Jul 24 22:47:51.891632 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:47:51.891598    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nvrzc\" is forbidden: User \"system:node:ip-10-0-129-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-cfjts\": no relationship found between node 'ip-10-0-129-163.ec2.internal' and this object" podUID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:52.316097 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.315400    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cfjts_must-gather-nvrzc_6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6/copy/0.log"
Jul 24 22:47:52.321714 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.321578    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:52.399850 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.399808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-must-gather-output\") pod \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\" (UID: \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\") "
Jul 24 22:47:52.400041 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.399883    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bmmb\" (UniqueName: \"kubernetes.io/projected/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-kube-api-access-2bmmb\") pod \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\" (UID: \"6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6\") "
Jul 24 22:47:52.402082 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.402045    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" (UID: "6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:47:52.414258 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.414212    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-kube-api-access-2bmmb" (OuterVolumeSpecName: "kube-api-access-2bmmb") pod "6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" (UID: "6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6"). InnerVolumeSpecName "kube-api-access-2bmmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:47:52.501126 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.501060    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-must-gather-output\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:47:52.501126 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.501098    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bmmb\" (UniqueName: \"kubernetes.io/projected/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6-kube-api-access-2bmmb\") on node \"ip-10-0-129-163.ec2.internal\" DevicePath \"\""
Jul 24 22:47:52.655780 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.655745    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-cfjts_must-gather-nvrzc_6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6/copy/0.log"
Jul 24 22:47:52.656165 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.656136    2580 generic.go:356] "Generic (PLEG): container finished" podID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" containerID="c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949" exitCode=143
Jul 24 22:47:52.656267 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.656207    2580 scope.go:117] "RemoveContainer" containerID="c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949"
Jul 24 22:47:52.656381 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.656364    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cfjts/must-gather-nvrzc"
Jul 24 22:47:52.672694 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.672667    2580 scope.go:117] "RemoveContainer" containerID="c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353"
Jul 24 22:47:52.715474 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.715439    2580 scope.go:117] "RemoveContainer" containerID="c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949"
Jul 24 22:47:52.715964 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:47:52.715925    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949\": container with ID starting with c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949 not found: ID does not exist" containerID="c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949"
Jul 24 22:47:52.716076 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.715967    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949"} err="failed to get container status \"c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949\": rpc error: code = NotFound desc = could not find container \"c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949\": container with ID starting with c1f66cdc0f94f606f1fa8ca0de507e4fdd1b4c2cc4a8a4af91d7bd86a1035949 not found: ID does not exist"
Jul 24 22:47:52.716076 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.715992    2580 scope.go:117] "RemoveContainer" containerID="c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353"
Jul 24 22:47:52.716377 ip-10-0-129-163 kubenswrapper[2580]: E0724 22:47:52.716339    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353\": container with ID starting with c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353 not found: ID does not exist" containerID="c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353"
Jul 24 22:47:52.716488 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:52.716373    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353"} err="failed to get container status \"c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353\": rpc error: code = NotFound desc = could not find container \"c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353\": container with ID starting with c5fd53fc1ef1cf71b75f8785ec3f82521a93ffebba2ea192dc1fb883938b2353 not found: ID does not exist"
Jul 24 22:47:54.290425 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.289909    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" path="/var/lib/kubelet/pods/6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6/volumes"
Jul 24 22:47:54.474614 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.474525    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c/alertmanager/0.log"
Jul 24 22:47:54.507678 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.507651    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c/config-reloader/0.log"
Jul 24 22:47:54.534558 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.534527    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c/kube-rbac-proxy-web/0.log"
Jul 24 22:47:54.562252 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.562220    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c/kube-rbac-proxy/0.log"
Jul 24 22:47:54.586598 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.586567    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c/kube-rbac-proxy-metric/0.log"
Jul 24 22:47:54.619461 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.619430    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c/prom-label-proxy/0.log"
Jul 24 22:47:54.653068 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.653033    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8b8a96b5-e5e0-4368-ad10-e0b1f9fbe16c/init-config-reloader/0.log"
Jul 24 22:47:54.740319 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.740295    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-7tc4r_2c41544f-1c45-4e29-a279-3451ab823866/kube-state-metrics/0.log"
Jul 24 22:47:54.761786 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.761760    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-7tc4r_2c41544f-1c45-4e29-a279-3451ab823866/kube-rbac-proxy-main/0.log"
Jul 24 22:47:54.786533 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.786494    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-7tc4r_2c41544f-1c45-4e29-a279-3451ab823866/kube-rbac-proxy-self/0.log"
Jul 24 22:47:54.820548 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.820516    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f54757957-kqcgb_2afef473-dc3f-422b-a6ab-b56afd71cf3c/metrics-server/0.log"
Jul 24 22:47:54.849785 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:54.849744    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-57d8p_d6fd70cf-883d-400a-8c9f-7c4ebe14d02f/monitoring-plugin/0.log"
Jul 24 22:47:55.053742 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.053711    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxdqv_af882602-2d6e-4174-9886-3029be000688/node-exporter/0.log"
Jul 24 22:47:55.081432 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.081408    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxdqv_af882602-2d6e-4174-9886-3029be000688/kube-rbac-proxy/0.log"
Jul 24 22:47:55.108001 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.107970    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxdqv_af882602-2d6e-4174-9886-3029be000688/init-textfile/0.log"
Jul 24 22:47:55.145036 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.145001    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mcrkx_b5ee866b-2902-4480-a99c-9aff3bebcac4/kube-rbac-proxy-main/0.log"
Jul 24 22:47:55.174848 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.174801    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mcrkx_b5ee866b-2902-4480-a99c-9aff3bebcac4/kube-rbac-proxy-self/0.log"
Jul 24 22:47:55.200436 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.200400    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mcrkx_b5ee866b-2902-4480-a99c-9aff3bebcac4/openshift-state-metrics/0.log"
Jul 24 22:47:55.247842 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.247809    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73735a04-4aaa-4d93-b216-8f8f9c019174/prometheus/0.log"
Jul 24 22:47:55.267332 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.267303    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73735a04-4aaa-4d93-b216-8f8f9c019174/config-reloader/0.log"
Jul 24 22:47:55.292144 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.292115    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73735a04-4aaa-4d93-b216-8f8f9c019174/thanos-sidecar/0.log"
Jul 24 22:47:55.313206 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.313133    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73735a04-4aaa-4d93-b216-8f8f9c019174/kube-rbac-proxy-web/0.log"
Jul 24 22:47:55.336461 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.336431    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73735a04-4aaa-4d93-b216-8f8f9c019174/kube-rbac-proxy/0.log"
Jul 24 22:47:55.362672 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.362644    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73735a04-4aaa-4d93-b216-8f8f9c019174/kube-rbac-proxy-thanos/0.log"
Jul 24 22:47:55.385734 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.385703    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73735a04-4aaa-4d93-b216-8f8f9c019174/init-config-reloader/0.log"
Jul 24 22:47:55.417338 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.417304    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-r7kxf_d1ac3543-e409-44c2-8e04-edd3a1fc2da9/prometheus-operator/0.log"
Jul 24 22:47:55.436379 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.436341    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-r7kxf_d1ac3543-e409-44c2-8e04-edd3a1fc2da9/kube-rbac-proxy/0.log"
Jul 24 22:47:55.458847 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.458807    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-2mq5z_2e641a9f-5b54-441c-94b4-bda51b4fcaed/prometheus-operator-admission-webhook/0.log"
Jul 24 22:47:55.490740 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.490684    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-576c5d977c-gtbzz_f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0/telemeter-client/0.log"
Jul 24 22:47:55.517067 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.517041    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-576c5d977c-gtbzz_f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0/reload/0.log"
Jul 24 22:47:55.544074 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.544048    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-576c5d977c-gtbzz_f0c96c61-0201-4d3a-ae0a-dedb7d9b22e0/kube-rbac-proxy/0.log"
Jul 24 22:47:55.581629 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.581536    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f6f4df5bf-d4psb_dc70df34-6665-4357-a4ed-89ea199ff679/thanos-query/0.log"
Jul 24 22:47:55.603885 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.603861    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f6f4df5bf-d4psb_dc70df34-6665-4357-a4ed-89ea199ff679/kube-rbac-proxy-web/0.log"
Jul 24 22:47:55.625320 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.625294    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f6f4df5bf-d4psb_dc70df34-6665-4357-a4ed-89ea199ff679/kube-rbac-proxy/0.log"
Jul 24 22:47:55.646750 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.646718    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f6f4df5bf-d4psb_dc70df34-6665-4357-a4ed-89ea199ff679/prom-label-proxy/0.log"
Jul 24 22:47:55.667914 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.667893    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f6f4df5bf-d4psb_dc70df34-6665-4357-a4ed-89ea199ff679/kube-rbac-proxy-rules/0.log"
Jul 24 22:47:55.688960 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:55.688933    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f6f4df5bf-d4psb_dc70df34-6665-4357-a4ed-89ea199ff679/kube-rbac-proxy-metrics/0.log"
Jul 24 22:47:57.949492 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:57.949437    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-nfqff_58895569-e4cb-4d1a-8d28-576d265b1c16/download-server/0.log"
Jul 24 22:47:58.366882 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.366842    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"]
Jul 24 22:47:58.367404 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.367361    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" containerName="gather"
Jul 24 22:47:58.368215 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.368198    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" containerName="gather"
Jul 24 22:47:58.368357 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.368343    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" containerName="copy"
Jul 24 22:47:58.368494 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.368482    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" containerName="copy"
Jul 24 22:47:58.368754 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.368722    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" containerName="gather"
Jul 24 22:47:58.368754 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.368748    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cf00eb8-1d65-44cf-83de-e1a5f8b2bff6" containerName="copy"
Jul 24 22:47:58.374313 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.374292    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.381037 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.380997    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"]
Jul 24 22:47:58.464839 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.464808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-proc\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.464839 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.464842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-podres\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.465062 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.464886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-lib-modules\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.465062 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.464943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-sys\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.465062 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.464961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m4ds\" (UniqueName: \"kubernetes.io/projected/070d227f-0c98-406d-b18a-705f4b31bbf4-kube-api-access-6m4ds\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.565796 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.565761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-lib-modules\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.565970 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.565835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-sys\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.565970 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.565862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4ds\" (UniqueName: \"kubernetes.io/projected/070d227f-0c98-406d-b18a-705f4b31bbf4-kube-api-access-6m4ds\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.565970 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.565890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-proc\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.565970 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.565913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-podres\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.565970 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.565944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-lib-modules\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.566162 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.565944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-sys\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.566162 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.566016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-podres\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.566162 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.566026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/070d227f-0c98-406d-b18a-705f4b31bbf4-proc\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.574627 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.574598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m4ds\" (UniqueName: \"kubernetes.io/projected/070d227f-0c98-406d-b18a-705f4b31bbf4-kube-api-access-6m4ds\") pod \"perf-node-gather-daemonset-djq7n\" (UID: \"070d227f-0c98-406d-b18a-705f4b31bbf4\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.690506 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.690421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:58.868840 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:58.868409    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"]
Jul 24 22:47:59.369189 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:59.369107    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5jb5d_3d659a21-4d00-4e07-8a91-8e1401c69d1c/dns/0.log"
Jul 24 22:47:59.390125 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:59.390100    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5jb5d_3d659a21-4d00-4e07-8a91-8e1401c69d1c/kube-rbac-proxy/0.log"
Jul 24 22:47:59.558517 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:59.558490    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zvggz_1328ac14-7eae-451f-af99-eb968a843e61/dns-node-resolver/0.log"
Jul 24 22:47:59.702292 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:59.702205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n" event={"ID":"070d227f-0c98-406d-b18a-705f4b31bbf4","Type":"ContainerStarted","Data":"a4156d49a6b33729d912c3309494b2db7f7ace293cd8193d8d3e96d7faceaaf9"}
Jul 24 22:47:59.702292 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:59.702265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n" event={"ID":"070d227f-0c98-406d-b18a-705f4b31bbf4","Type":"ContainerStarted","Data":"ab337d9d35c5043f17df51150f9adf6c79992130568d9c02d080b148adaa825a"}
Jul 24 22:47:59.702292 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:59.702290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:47:59.720642 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:47:59.720590    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n" podStartSLOduration=1.7205732230000002 podStartE2EDuration="1.720573223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:59.719328198 +0000 UTC m=+5554.058169974" watchObservedRunningTime="2026-07-24 22:47:59.720573223 +0000 UTC m=+5554.059414999"
Jul 24 22:48:00.098408 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:00.098368    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qtpfl_6bf75a47-f6ab-4c0e-bf43-3e2474e2ca54/node-ca/0.log"
Jul 24 22:48:01.323890 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:01.323866    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zhzpr_a88f4310-55f1-4b9e-9cd8-f879159e4818/serve-healthcheck-canary/0.log"
Jul 24 22:48:01.746246 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:01.746160    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-stc25_cd2c52d4-5c41-45fc-9d06-39a03c1a80cb/kube-rbac-proxy/0.log"
Jul 24 22:48:01.767137 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:01.767108    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-stc25_cd2c52d4-5c41-45fc-9d06-39a03c1a80cb/exporter/0.log"
Jul 24 22:48:01.789216 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:01.789186    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-stc25_cd2c52d4-5c41-45fc-9d06-39a03c1a80cb/extractor/0.log"
Jul 24 22:48:04.188522 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:04.188497    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-smgwt_87c07919-5eaa-4979-a6fe-5a9c6b41d882/server/0.log"
Jul 24 22:48:04.533034 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:04.532999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4dq9z_d63df2c9-0e80-449d-9fa8-9945622619f6/s3-init/0.log"
Jul 24 22:48:04.549360 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:04.549337    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4dq9z_d63df2c9-0e80-449d-9fa8-9945622619f6/download-hf-model/0.log"
Jul 24 22:48:04.584247 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:04.584221    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-sbgpg_5df498cd-676f-407e-9833-68d40b1656a9/s3-tls-init-custom/0.log"
Jul 24 22:48:04.599913 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:04.599889    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-sbgpg_5df498cd-676f-407e-9833-68d40b1656a9/download-hf-model/0.log"
Jul 24 22:48:04.633101 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:04.633064    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4tcvm_8dfff6e5-ea64-4a5e-a317-afb411385378/s3-tls-init-serving/0.log"
Jul 24 22:48:04.649259 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:04.649226    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-4tcvm_8dfff6e5-ea64-4a5e-a317-afb411385378/download-hf-model/0.log"
Jul 24 22:48:04.678102 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:04.678052    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-b8v9m_5cae5431-2d94-44fb-8a3b-0fac7974678c/seaweedfs/0.log"
Jul 24 22:48:04.704353 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:04.704318    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-8m2rk_d9d923c4-9c10-4a98-9abd-02fabb376f17/seaweedfs-tls-custom/0.log"
Jul 24 22:48:05.718283 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:05.718259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-djq7n"
Jul 24 22:48:10.365577 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.365546    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgqmz_4e1f81e2-e886-4bbf-9b1c-e4d70201b449/kube-multus-additional-cni-plugins/0.log"
Jul 24 22:48:10.385785 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.385754    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgqmz_4e1f81e2-e886-4bbf-9b1c-e4d70201b449/egress-router-binary-copy/0.log"
Jul 24 22:48:10.405521 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.405479    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgqmz_4e1f81e2-e886-4bbf-9b1c-e4d70201b449/cni-plugins/0.log"
Jul 24 22:48:10.425500 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.425477    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgqmz_4e1f81e2-e886-4bbf-9b1c-e4d70201b449/bond-cni-plugin/0.log"
Jul 24 22:48:10.445730 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.445707    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgqmz_4e1f81e2-e886-4bbf-9b1c-e4d70201b449/routeoverride-cni/0.log"
Jul 24 22:48:10.465495 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.465469    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgqmz_4e1f81e2-e886-4bbf-9b1c-e4d70201b449/whereabouts-cni-bincopy/0.log"
Jul 24 22:48:10.487202 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.487180    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cgqmz_4e1f81e2-e886-4bbf-9b1c-e4d70201b449/whereabouts-cni/0.log"
Jul 24 22:48:10.737896 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.737818    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q6tdh_eb587855-58e3-4504-83e8-71afb6f79cba/kube-multus/0.log"
Jul 24 22:48:10.810856 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.810822    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-njtn4_514cfda0-541e-4b7d-91dd-dcb870c17692/network-metrics-daemon/0.log"
Jul 24 22:48:10.827238 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:10.827215    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-njtn4_514cfda0-541e-4b7d-91dd-dcb870c17692/kube-rbac-proxy/0.log"
Jul 24 22:48:11.795875 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:11.795848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-controller/0.log"
Jul 24 22:48:11.812824 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:11.812765    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/0.log"
Jul 24 22:48:11.813100 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:11.812821    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovn-acl-logging/1.log"
Jul 24 22:48:11.830689 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:11.830670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/kube-rbac-proxy-node/0.log"
Jul 24 22:48:11.850623 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:11.850600    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/kube-rbac-proxy-ovn-metrics/0.log"
Jul 24 22:48:11.869087 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:11.869057    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/northd/0.log"
Jul 24 22:48:11.889534 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:11.889513    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/nbdb/0.log"
Jul 24 22:48:11.911407 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:11.911368    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/sbdb/0.log"
Jul 24 22:48:11.987229 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:11.987199    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d5rxk_ff698d8b-a442-4540-9084-615541d3bcff/ovnkube-controller/0.log"
Jul 24 22:48:13.139345 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:13.139308    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8vg5v_d6bcfcfc-b6c6-42d3-8914-9a1db1edfde0/network-check-target-container/0.log"
Jul 24 22:48:14.100430 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:14.100404    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-pbjmp_14ca3ba3-0b3a-4063-9f59-e4f205927cfe/iptables-alerter/0.log"
Jul 24 22:48:14.720515 ip-10-0-129-163 kubenswrapper[2580]: I0724 22:48:14.720487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-cjvwv_2bb3d2d5-8956-4fc5-b3e8-abb97c415f41/tuned/0.log"