Jul 27 17:25:29.051552 ip-10-0-129-58 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 27 17:25:29.051721 ip-10-0-129-58 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 27 17:25:29.051870 ip-10-0-129-58 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 27 17:25:29.052238 ip-10-0-129-58 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 27 17:25:39.197499 ip-10-0-129-58 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 27 17:25:39.197513 ip-10-0-129-58 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bbf4c387cc2c45838e457bdf5b26aa1b --
Jul 27 17:27:36.432510 ip-10-0-129-58 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 17:27:36.826916 ip-10-0-129-58 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:27:36.826916 ip-10-0-129-58 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 17:27:36.826916 ip-10-0-129-58 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:27:36.826916 ip-10-0-129-58 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 17:27:36.826916 ip-10-0-129-58 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:27:36.828444 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.828360    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 17:27:36.835398 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835382    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:36.835398 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835398    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835402    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835407    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835410    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835414    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835417    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835420    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835423    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835425    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835429    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835432    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835435    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835438    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835440    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835443    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835446    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835449    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835452    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835454    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835457    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:36.835469 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835460    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835463    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835465    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835468    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835470    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835473    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835476    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835479    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835482    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835484    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835487    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835489    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835492    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835495    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835497    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835499    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835502    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835504    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835507    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835510    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:36.835938 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835512    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835515    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835518    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835521    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835523    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835526    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835529    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835531    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835534    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835536    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835539    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835541    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835543    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835546    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835550    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835553    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835556    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835558    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835560    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835564    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:36.836441 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835566    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835569    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835581    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835584    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835587    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835589    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835592    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835594    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835597    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835600    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835602    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835605    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835607    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835611    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835615    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835618    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835621    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835624    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835627    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:36.836964 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835629    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835634    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835638    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835641    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835645    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835648    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835651    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835654    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835657    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835660    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835663    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835666    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835668    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835671    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835674    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835676    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835679    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835682    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.835684    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:36.837442 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836420    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836427    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836431    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836434    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836437    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836440    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836443    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836446    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836449    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836452    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836455    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836458    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836461    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836464    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836467    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836470    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836472    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836475    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836477    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836480    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:36.837896 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836483    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836486    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836488    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836491    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836493    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836496    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836498    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836501    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836503    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836505    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836508    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836510    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836513    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836515    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836518    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836522    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836525    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836529    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836531    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:36.838392 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836534    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836536    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836539    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836542    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836546    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836549    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836552    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836555    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836557    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836560    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836563    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836565    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836568    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836571    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836573    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836576    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836579    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836582    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836584    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:36.838855 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836586    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836589    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836591    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836594    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836596    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836598    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836601    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836603    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836606    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836608    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836611    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836613    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836616    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836619    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836621    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836624    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836626    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836630    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836633    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836635    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:36.839365 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836638    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836640    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836643    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836645    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836647    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836650    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836654    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836657    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836660    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836662    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836665    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836668    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836670    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836673    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836675    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836678    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836680    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836683    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836685    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836688    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:36.839845 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.836692    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837047    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837051    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837054    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837057    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837060    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837063    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837065    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837069    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837077    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837081    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837084    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837086    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837089    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837091    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837094    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837097    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837099    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837102    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:36.840348 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837105    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837107    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837110    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837112    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837115    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837118    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837121    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837123    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837126    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837128    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837131    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837134    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837136    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837138    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837142    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837145    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837148    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837151    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837154    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837156    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:36.840804 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837159    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837161    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837164    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837173    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837175    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837178    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837197    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837199    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837202    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837205    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837207    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837210    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837213    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837216    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837218    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837221    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837223    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837226    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837229    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837231    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:36.841300 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837234    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837236    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837239    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837241    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837244    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837246    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837249    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837251    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837255    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837259    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837262    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837265    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837275    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837278    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837281    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837284    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837292    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837295    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837298    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:36.841795 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837300    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837303    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837306    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837309    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837312    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837316    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837319    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837329    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837333    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837335    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837338    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837342    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837345    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837348    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837351    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837354    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837357    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837359    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837362    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837364    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:36.842268 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837367    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.837370    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837444    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837451    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837460    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837467    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837473    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837478    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837483    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837487    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837491    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837494    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837498    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837501    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837504    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837507    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837513    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837516    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837519    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837522    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837526    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837529    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837533    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837536    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 17:27:36.842831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837539    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837542    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837546    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837549    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837552    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837555    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837558    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837561    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837564    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837566    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837571    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837574    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837577    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837580    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837582    2564 flags.go:64] FLAG: --enable-server="true"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837585    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837590    2564 flags.go:64] FLAG: --event-burst="100"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837593    2564 flags.go:64] FLAG: --event-qps="50"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837596    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837599    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837602    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837606    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837611    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837614    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837618    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 27 17:27:36.843407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837621    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837624    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837627    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837630    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837633    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837636    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837639    2564 flags.go:64] FLAG: --feature-gates=""
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837642    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837645    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837648    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837651    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837654    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837657    2564 flags.go:64] FLAG: --help="false"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837660    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-129-58.ec2.internal"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837664    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837666    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837669    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837673    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837676    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837679    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837682    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837685    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837687    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 17:27:36.844022 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837690    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837693    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837696    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837699    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837702    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837705    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837710    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837713    2564 flags.go:64] FLAG: --lock-file=""
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837716    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837719    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837722    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837727    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837730    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837733    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837736    2564 flags.go:64] FLAG: --logging-format="text"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837739    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837742    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837745    2564 flags.go:64] FLAG: --manifest-url=""
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837748    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837752    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837755    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837759    2564 flags.go:64] FLAG: --max-pods="110"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837762    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837765    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837767    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 17:27:36.844617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837770    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837773    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837776    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837779    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837786    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837790    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837793    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837796    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837799    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837804    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837807    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837822    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837828    2564 flags.go:64] FLAG: --port="10250"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837832    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837836    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04e47c160428277cd"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837839    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837844    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837847    2564 flags.go:64] FLAG: --register-node="true"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837850    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837854    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837857    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837859    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837862    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837866    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837869    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 17:27:36.845252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837872    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837875    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837878    2564 flags.go:64] FLAG: --runonce="false"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837881    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837884    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837887    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837890    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837892    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837896    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837899    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837902    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837905    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837908    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837911    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837914    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837917    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837920    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837923    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837928    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837931    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837936    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837940    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837943    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837947    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837950    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 17:27:36.845864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837953    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837956    2564 flags.go:64] FLAG: --v="2"
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837960    2564 flags.go:64] FLAG: --version="false"
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837964    2564 flags.go:64] FLAG: --vmodule=""
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837968    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.837971    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838088    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838092    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838095    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838098    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838100    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838103    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838106    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838109    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838111    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838114    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838116    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838119    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838123    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838125    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838129    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:36.846479 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838133    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838136    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838138    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838141    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838144    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838146    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838150    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838153    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838156    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838160    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838162    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838165    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838167    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838170    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838172    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838175    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838178    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838193    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838196    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838198    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:36.846972 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838201    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838204    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838206    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838209    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838211    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838214    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838217    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838220    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838222    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838225    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838228    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838230    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838233    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838235    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838238    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838241    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838243    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838246    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838250    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:36.847468 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838252    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838255    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838259    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838262    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838265    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838267    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838270    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838272    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838275    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838278    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838280    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838283    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838285    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838288    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838291    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838293    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838296    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838300    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838303    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838306    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:36.847962 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838309    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838312    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838314    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838318    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838320    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838323    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838326    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838328    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838331    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838333    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838336    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838339    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838343    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838347    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838352    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838355    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838358    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838360    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838363    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:36.848489 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838366    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838368    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838370    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838373    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838376    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.838378    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.838990    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.846066    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.846087    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.846107    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.846112    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846194    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846201    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846205    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:36.848940 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846210    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846214    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846218    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846221    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846224    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846228    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846231    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846233    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846237    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846240    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846243    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846245    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846248    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846251    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846253    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846256    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846258    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846261    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846263    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846266    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:36.849298 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846269    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846271    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846274    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846276    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846279    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846281    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846299    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846303    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846306    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846308    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846311    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846314    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846316    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846319    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846321    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846324    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846326    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846329    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846332    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846334    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:36.849825 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846337    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846339    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846342    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846344    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846347    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846349    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846352    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846355    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846357    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846360    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846362    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846366    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846369    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846372    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846378    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846381    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846383    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846387    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846391    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846399    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:36.850677 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846402    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846404    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846407    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846410    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846412    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846415    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846417    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846420    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846423    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846425    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846428    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846431    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846434    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846436    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846439    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846441    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846444    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846447    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846449    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846452    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:36.851440 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846454    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846457    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846459    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846462    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846464    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846467    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846469    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846472    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846475    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846477    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846480    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846483    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846491    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846494    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846497    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:36.851936 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846499    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.846504    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846694    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846700    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846703    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846706    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846708    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846711    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846713    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846716    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846719    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846721    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846724    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846726    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:36.852370 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846729    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846732    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846735    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846737    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846740    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846744    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846748    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846751    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846754    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846757    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846760    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846764    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846767    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846771    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846773    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846776    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846785    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846788    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846791    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:36.852731 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846794    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846797    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846799    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846802    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846805    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846807    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846810    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846812    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846815    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846817    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846820    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846822    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846825    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846828    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846830    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846832    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846835    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846838    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846840    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846843    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:36.853220 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846845    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846847    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846850    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846853    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846856    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846858    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846861    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846863    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846866    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846868    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846876    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846879    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846881    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846884    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846886    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846889    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846891    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846894    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846896    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846899    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:36.853735 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846901    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846903    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846907    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846910    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846913    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846916    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846918    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846921    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846923    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846926    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846928    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846931    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846933    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846936    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846939    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846941    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846944    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846947    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846949    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846952    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:36.854232 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846954    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846957    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846959    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846967    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846970    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846973    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846975    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:36.846978    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.846983    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.847121    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.849084    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.851280    2564 server.go:1019] "Starting client certificate rotation"
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.851398    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:27:36.854726 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.851447    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:27:36.855049 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.855040    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:27:36.857103 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.857087    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:27:36.868787 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.868766    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 27 17:27:36.871141 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.871125    2564 log.go:25] "Validated CRI v1 image API"
Jul 27 17:27:36.872012 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.871987    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 17:27:36.874681 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.874660    2564 fs.go:135] Filesystem UUIDs: map[3bc01f1a-7984-4633-a91d-733b3266fd1a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cc9fe3b7-c42c-4ddf-a9e8-2709d66be65c:/dev/nvme0n1p4]
Jul 27 17:27:36.874747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.874678    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 17:27:36.878477 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.878457    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:27:36.880749 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.880629    2564 manager.go:216] Machine: {Timestamp:2026-07-27 17:27:36.879309112 +0000 UTC m=+0.331930269 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107269 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f495442976f434963832faeeaf8ee SystemUUID:ec2f4954-4297-6f43-4963-832faeeaf8ee BootID:bbf4c387-cc2c-4583-8e45-7bdf5b26aa1b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:00:e3:15:bf:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:00:e3:15:bf:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:a2:dc:0c:ac:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 17:27:36.880749 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.880740    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 17:27:36.880883 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.880826    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 17:27:36.883015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.882985    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 17:27:36.883162 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.883018    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-58.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 17:27:36.883776 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.883766    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 17:27:36.883806 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.883778    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 17:27:36.883806 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.883786    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:27:36.884402 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.884391    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:27:36.885058 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.885048    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 17:27:36.886684 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.886672    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:27:36.886811 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.886800    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 17:27:36.886848 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.886838    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 17:27:36.886878 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.886849    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 17:27:36.886878 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.886860    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 17:27:36.886878 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.886869    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 27 17:27:36.886953 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.886885    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 17:27:36.888069 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.888052    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:27:36.888170 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.888075    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:27:36.890774 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.890756    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 17:27:36.892662 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.892642    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 17:27:36.892729 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.892675    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 17:27:36.893828 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893817    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 17:27:36.893874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893832    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 17:27:36.893874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893837    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 17:27:36.893874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893842    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 17:27:36.893874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893850    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 17:27:36.893874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893855    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 17:27:36.893874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893863    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 17:27:36.893874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893868    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 17:27:36.893874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893874    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 17:27:36.894062 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893879    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 17:27:36.894062 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893887    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 17:27:36.894062 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.893892    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 17:27:36.894622 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.894613    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 17:27:36.894622 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.894622    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 17:27:36.896988 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.896970    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 17:27:36.897060 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.896988    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-58.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 17:27:36.897255 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.897231    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 17:27:36.897545 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.897535    2564 server.go:1268] "Started kubelet"
Jul 27 17:27:36.897614 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.897586    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 17:27:36.897709 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.897629    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 17:27:36.897709 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.897691    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 17:27:36.898286 ip-10-0-129-58 systemd[1]: Started Kubernetes Kubelet.
Jul 27 17:27:36.898924 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.898879    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 17:27:36.899142 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.899131    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 17:27:36.902850 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.902824    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 17:27:36.903262 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.902038    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-58.ec2.internal.18c6359e6fe74529  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-58.ec2.internal,UID:ip-10-0-129-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-58.ec2.internal,},FirstTimestamp:2026-07-27 17:27:36.897512745 +0000 UTC m=+0.350133878,LastTimestamp:2026-07-27 17:27:36.897512745 +0000 UTC m=+0.350133878,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-58.ec2.internal,}"
Jul 27 17:27:36.905284 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.905268    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 17:27:36.905284 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.905275    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 17:27:36.905405 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.905343    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:36.905776 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.905759    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 17:27:36.905855 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.905797    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 17:27:36.905855 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.905812    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 17:27:36.905952 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.905884    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 17:27:36.905952 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.905893    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 17:27:36.905952 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.905907    2564 factory.go:55] Registering systemd factory
Jul 27 17:27:36.905952 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.905923    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 27 17:27:36.906107 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.906097    2564 factory.go:153] Registering CRI-O factory
Jul 27 17:27:36.906159 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.906109    2564 factory.go:223] Registration of the crio container factory successfully
Jul 27 17:27:36.906231 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.906161    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 17:27:36.906231 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.906198    2564 factory.go:103] Registering Raw factory
Jul 27 17:27:36.906231 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.906216    2564 manager.go:1196] Started watching for new ooms in manager
Jul 27 17:27:36.906515 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.906502    2564 manager.go:349] Starting recovery of all containers
Jul 27 17:27:36.907491 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.907469    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 17:27:36.907580 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.907563    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 17:27:36.916442 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.916427    2564 manager.go:354] Recovery completed
Jul 27 17:27:36.920447 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.920435    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:36.921936 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.921911    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:36.921997 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.921948    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:36.921997 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.921957    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:36.922520 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.922504    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 17:27:36.922592 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.922519    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 17:27:36.922592 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.922529    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:36.922592 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.922536    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:27:36.923149 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.923135    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:36.923257 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.923159    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:36.923257 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.923170    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:36.924156 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.924145    2564 policy_none.go:49] "None policy: Start"
Jul 27 17:27:36.924215 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.924164    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 17:27:36.924215 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.924191    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 17:27:36.925642 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.925633    2564 policy_none.go:47] "Start"
Jul 27 17:27:36.926800 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.926786    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-57b8f"
Jul 27 17:27:36.931841 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.931825    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-57b8f"
Jul 27 17:27:36.932525 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.932453    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-58.ec2.internal.18c6359e715bf273  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-58.ec2.internal,UID:ip-10-0-129-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-58.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-58.ec2.internal,},FirstTimestamp:2026-07-27 17:27:36.921936499 +0000 UTC m=+0.374557633,LastTimestamp:2026-07-27 17:27:36.921936499 +0000 UTC m=+0.374557633,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-58.ec2.internal,}"
Jul 27 17:27:36.954829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.954720    2564 manager.go:337] "Starting Device Plugin manager"
Jul 27 17:27:36.954926 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.954851    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 17:27:36.954926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.954864    2564 server.go:87] "Starting device plugin registration server"
Jul 27 17:27:36.955082 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.955071    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 17:27:36.955141 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.955083    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 17:27:36.955297 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.955277    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 17:27:36.955377 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.955355    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 17:27:36.955377 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:36.955362    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 17:27:36.955847 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.955831    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 17:27:36.955904 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:36.955869    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.055379 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.055352    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:37.056306 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.056204    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:37.056306 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.056232    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:37.056306 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.056243    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:37.056306 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.056269    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.064938 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.064921    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.065034 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.064946    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-58.ec2.internal\": node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.088990 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.088914    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.112717 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.112682    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 17:27:37.113931 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.113911    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 17:27:37.114014 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.113948    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 17:27:37.114014 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.113967    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 17:27:37.114014 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.113997    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 17:27:37.116212 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.116196    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:37.189266 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.189231    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.214583 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.214555    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"]
Jul 27 17:27:37.214660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.214621    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:37.216174 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.216151    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:37.216274 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.216203    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:37.216274 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.216218    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:37.217304 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.217292    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:37.217403 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.217380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.217448 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.217415    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:37.218016 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.218003    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:37.218086 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.218030    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:37.218086 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.218045    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:37.218086 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.218007    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:37.218237 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.218093    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:37.218237 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.218103    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:37.219079 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.219066    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.219128 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.219090    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:37.221913 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.221895    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:37.221970 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.221928    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:37.221970 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.221943    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:37.234674 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.234658    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-58.ec2.internal\" not found" node="ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.238641 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.238626    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-58.ec2.internal\" not found" node="ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.289920 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.289890    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.308197 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.308156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebc1b9d3588ffefd5cf60f36ae3035ca-config\") pod \"kube-apiserver-proxy-ip-10-0-129-58.ec2.internal\" (UID: \"ebc1b9d3588ffefd5cf60f36ae3035ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.308325 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.308203    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3fecf50f073a1a78674a6ab671580e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal\" (UID: \"d3fecf50f073a1a78674a6ab671580e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.308325 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.308222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3fecf50f073a1a78674a6ab671580e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal\" (UID: \"d3fecf50f073a1a78674a6ab671580e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.390483 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.390421    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.408815 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.408788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3fecf50f073a1a78674a6ab671580e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal\" (UID: \"d3fecf50f073a1a78674a6ab671580e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.408885 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.408820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebc1b9d3588ffefd5cf60f36ae3035ca-config\") pod \"kube-apiserver-proxy-ip-10-0-129-58.ec2.internal\" (UID: \"ebc1b9d3588ffefd5cf60f36ae3035ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.408885 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.408838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3fecf50f073a1a78674a6ab671580e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal\" (UID: \"d3fecf50f073a1a78674a6ab671580e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.408976 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.408880    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3fecf50f073a1a78674a6ab671580e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal\" (UID: \"d3fecf50f073a1a78674a6ab671580e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.408976 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.408883    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebc1b9d3588ffefd5cf60f36ae3035ca-config\") pod \"kube-apiserver-proxy-ip-10-0-129-58.ec2.internal\" (UID: \"ebc1b9d3588ffefd5cf60f36ae3035ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.408976 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.408889    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3fecf50f073a1a78674a6ab671580e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal\" (UID: \"d3fecf50f073a1a78674a6ab671580e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.490887 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.490861    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.537284 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.537256    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.541685 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.541669    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:37.591768 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.591743    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.692165 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.692097    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.778131 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.778107    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:37.792207 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.792176    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.850473 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.850453    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 17:27:37.850831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.850582    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 17:27:37.850831 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.850599    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 17:27:37.892842 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.892821    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:37.935809 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.935766    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 17:22:36 +0000 UTC" deadline="2028-02-01 20:57:08.219871817 +0000 UTC"
Jul 27 17:27:37.935809 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:37.935804    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13299h29m30.284071383s"
Jul 27 17:27:37.993022 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:37.993003    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-58.ec2.internal\" not found"
Jul 27 17:27:38.061712 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.061695    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:38.106306 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.106286    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:38.115497 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.115474    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:27:38.120974 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.120954    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"
Jul 27 17:27:38.133431 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.133413    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:27:38.166219 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.166196    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:38.219488 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:38.219453    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc1b9d3588ffefd5cf60f36ae3035ca.slice/crio-812573200e857b8bed307c2421d47aaf1ccdd0be1ab78b2a11be796a761c6061 WatchSource:0}: Error finding container 812573200e857b8bed307c2421d47aaf1ccdd0be1ab78b2a11be796a761c6061: Status 404 returned error can't find the container with id 812573200e857b8bed307c2421d47aaf1ccdd0be1ab78b2a11be796a761c6061
Jul 27 17:27:38.219893 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:38.219864    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fecf50f073a1a78674a6ab671580e5.slice/crio-16e28ed08563bfa8206c25f0fcb694c844137bb2dadab07a5950d387e392dfe3 WatchSource:0}: Error finding container 16e28ed08563bfa8206c25f0fcb694c844137bb2dadab07a5950d387e392dfe3: Status 404 returned error can't find the container with id 16e28ed08563bfa8206c25f0fcb694c844137bb2dadab07a5950d387e392dfe3
Jul 27 17:27:38.224103 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.224082    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:27:38.888908 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.888729    2564 apiserver.go:52] "Watching apiserver"
Jul 27 17:27:38.897432 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.897411    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 17:27:38.899285 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.899261    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gqmhw","openshift-multus/network-metrics-daemon-flnfk","openshift-network-diagnostics/network-check-target-8wtkw","openshift-network-operator/iptables-alerter-99rcs","openshift-ovn-kubernetes/ovnkube-node-fm8gt","kube-system/global-pull-secret-syncer-ggttg","kube-system/konnectivity-agent-fl5bn","kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g","openshift-cluster-node-tuning-operator/tuned-26rmm","openshift-dns/node-resolver-2lkcp","openshift-image-registry/node-ca-v8rh6","openshift-multus/multus-additional-cni-plugins-n252c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal"]
Jul 27 17:27:38.900929 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.900908    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:38.901019 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:38.900992    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:38.902238 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.902217    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.903359 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.903339    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:27:38.905105 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.905086    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:38.906017 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.905616    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 17:27:38.906017 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.905769    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 17:27:38.906017 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.905847    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 17:27:38.906017 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.905859    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 17:27:38.906279 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.906060    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 17:27:38.906385 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.906364    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 17:27:38.906442 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.906423    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 17:27:38.906943 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.906921    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 17:27:38.907019 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.906987    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n22gm\""
Jul 27 17:27:38.907120 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.906936    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d9lxx\""
Jul 27 17:27:38.907341 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.907218    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 17:27:38.907475 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.907452    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:38.908821 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:38.907514    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:38.908821 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.907983    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2mm7r\""
Jul 27 17:27:38.908821 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.908021    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 17:27:38.909015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.908947    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:27:38.909908 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.909887    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 17:27:38.910935 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.910919    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:38.911170 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:38.911149    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:38.913292 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913269    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:38.913784 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913760    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-run-systemd\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.913855 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-cni-bin\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.913855 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1d3ea62-7e04-4d73-a171-6e573d4e6637-ovnkube-config\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.913855 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1d3ea62-7e04-4d73-a171-6e573d4e6637-ovn-node-metrics-cert\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1d3ea62-7e04-4d73-a171-6e573d4e6637-ovnkube-script-lib\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e-host-slash\") pod \"iptables-alerter-99rcs\" (UID: \"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e\") " pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:38.914013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-kubelet\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-run-netns\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-var-lib-openvswitch\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.913988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-run-openvswitch\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-log-socket\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914032    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b95fb06-43f5-441b-b46c-60a5c2cc0d9e-agent-certs\") pod \"konnectivity-agent-fl5bn\" (UID: \"0b95fb06-43f5-441b-b46c-60a5c2cc0d9e\") " pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914079    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-slash\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914101    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-run-ovn\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-cni-netd\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914145    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-node-log\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914168    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914208    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9dw\" (UniqueName: \"kubernetes.io/projected/b1d3ea62-7e04-4d73-a171-6e573d4e6637-kube-api-access-nv9dw\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e-iptables-alerter-script\") pod \"iptables-alerter-99rcs\" (UID: \"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e\") " pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-etc-openvswitch\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjq8q\" (UniqueName: \"kubernetes.io/projected/a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e-kube-api-access-sjq8q\") pod \"iptables-alerter-99rcs\" (UID: \"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e\") " pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-systemd-units\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-run-ovn-kubernetes\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.914380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1d3ea62-7e04-4d73-a171-6e573d4e6637-env-overrides\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:38.915114 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914396    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b95fb06-43f5-441b-b46c-60a5c2cc0d9e-konnectivity-ca\") pod \"konnectivity-agent-fl5bn\" (UID: \"0b95fb06-43f5-441b-b46c-60a5c2cc0d9e\") " pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:27:38.915114 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwj6\" (UniqueName: \"kubernetes.io/projected/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-kube-api-access-xcwj6\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:38.915114 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.914790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:38.916221 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.916199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:38.916546 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.916525    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 17:27:38.916630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.916593    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 17:27:38.916630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.916529    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 17:27:38.916773 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.916531    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-95v77\""
Jul 27 17:27:38.917166 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.916973    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 17:27:38.917571 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.917552    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cjq9r\""
Jul 27 17:27:38.917803 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.917784    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 17:27:38.917870 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.917809    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 17:27:38.918019 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.918000    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:38.918124 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.918031    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 17:27:38.920048 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.918701    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-whdr9\""
Jul 27 17:27:38.920048 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.918783    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 17:27:38.920048 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.919024    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:27:38.920314 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.920297    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:38.920845 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.920582    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 17:27:38.921303 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.921278    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lf44s\""
Jul 27 17:27:38.921394 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.921307    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 17:27:38.921615 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.921594    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:38.922453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.922201    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:27:38.922858 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.922752    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 17:27:38.923481 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.923461    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 17:27:38.923748 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.923728    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 17:27:38.924563 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.924461    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 17:27:38.924744 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.924725    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 17:27:38.925425 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.925107    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wxqc9\""
Jul 27 17:27:38.925425 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.925374    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rpqnb\""
Jul 27 17:27:38.944970 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.944946    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-prkjc"
Jul 27 17:27:38.952744 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.952025    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-prkjc"
Jul 27 17:27:38.955736 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:38.955718    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:39.007983 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.007961    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 17:27:39.015327 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-run-ovn\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.015440 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-cni-netd\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.015440 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-etc-selinux\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.015440 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015398    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-sys-fs\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.015440 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015419    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-modprobe-d\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.015630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015444    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-sys\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.015630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-node-log\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.015630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e-iptables-alerter-script\") pod \"iptables-alerter-99rcs\" (UID: \"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e\") " pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:39.015630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-system-cni-dir\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.015630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.015630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-etc-kubernetes\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.015630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015592    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.015630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015618    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpcr\" (UniqueName: \"kubernetes.io/projected/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-kube-api-access-rgpcr\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015642    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-cni-dir\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-var-lib-cni-multus\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015688    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-hostroot\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-daemon-config\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-etc-openvswitch\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vklt\" (UniqueName: \"kubernetes.io/projected/a79b2ef9-8af0-49f3-8796-47f1db39e906-kube-api-access-5vklt\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-cnibin\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015827    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-cnibin\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-run-k8s-cni-cncf-io\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-host\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5qg2\" (UniqueName: \"kubernetes.io/projected/9c64d459-de22-4e66-a489-4c7ac0a62897-kube-api-access-f5qg2\") pod \"node-ca-v8rh6\" (UID: \"9c64d459-de22-4e66-a489-4c7ac0a62897\") " pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-socket-dir-parent\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015946    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-sysconfig\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-sysctl-d\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.016002 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.015994    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lq9v\" (UniqueName: \"kubernetes.io/projected/08be5885-b32c-4799-8157-d0c0000bf3ae-kube-api-access-4lq9v\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016026    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1d3ea62-7e04-4d73-a171-6e573d4e6637-env-overrides\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b95fb06-43f5-441b-b46c-60a5c2cc0d9e-konnectivity-ca\") pod \"konnectivity-agent-fl5bn\" (UID: \"0b95fb06-43f5-441b-b46c-60a5c2cc0d9e\") " pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwj6\" (UniqueName: \"kubernetes.io/projected/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-kube-api-access-xcwj6\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-registration-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c64d459-de22-4e66-a489-4c7ac0a62897-serviceca\") pod \"node-ca-v8rh6\" (UID: \"9c64d459-de22-4e66-a489-4c7ac0a62897\") " pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-cni-binary-copy\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-kubernetes\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-run-systemd\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1d3ea62-7e04-4d73-a171-6e573d4e6637-ovnkube-config\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016279    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e-host-slash\") pod \"iptables-alerter-99rcs\" (UID: \"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e\") " pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-device-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-run-netns\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-lib-modules\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016386    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-kubelet\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-run-netns\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b95fb06-43f5-441b-b46c-60a5c2cc0d9e-agent-certs\") pod \"konnectivity-agent-fl5bn\" (UID: \"0b95fb06-43f5-441b-b46c-60a5c2cc0d9e\") " pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:27:39.016761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08be5885-b32c-4799-8157-d0c0000bf3ae-tmp\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016486    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-system-cni-dir\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016522    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-socket-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c64d459-de22-4e66-a489-4c7ac0a62897-host\") pod \"node-ca-v8rh6\" (UID: \"9c64d459-de22-4e66-a489-4c7ac0a62897\") " pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016574    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9dw\" (UniqueName: \"kubernetes.io/projected/b1d3ea62-7e04-4d73-a171-6e573d4e6637-kube-api-access-nv9dw\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016660    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs8ms\" (UniqueName: \"kubernetes.io/projected/df9a0647-8600-491a-8042-186017147267-kube-api-access-vs8ms\") pod \"node-resolver-2lkcp\" (UID: \"df9a0647-8600-491a-8042-186017147267\") " pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-sysctl-conf\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df9a0647-8600-491a-8042-186017147267-hosts-file\") pod \"node-resolver-2lkcp\" (UID: \"df9a0647-8600-491a-8042-186017147267\") " pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-var-lib-cni-bin\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016765    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b376d75-6fb9-48ea-8621-e43b18a91e98-kubelet-config\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b376d75-6fb9-48ea-8621-e43b18a91e98-dbus\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016870    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-var-lib-kubelet\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.017588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-var-lib-kubelet\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-tuned\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjq8q\" (UniqueName: \"kubernetes.io/projected/a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e-kube-api-access-sjq8q\") pod \"iptables-alerter-99rcs\" (UID: \"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e\") " pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.016979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8024a66-c645-46d6-b1c3-f75dd18abb20-cni-binary-copy\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-run-multus-certs\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-systemd-units\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-run-ovn-kubernetes\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-cni-bin\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1d3ea62-7e04-4d73-a171-6e573d4e6637-ovn-node-metrics-cert\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017152    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1d3ea62-7e04-4d73-a171-6e573d4e6637-ovnkube-script-lib\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9a0647-8600-491a-8042-186017147267-tmp-dir\") pod \"node-resolver-2lkcp\" (UID: \"df9a0647-8600-491a-8042-186017147267\") " pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-conf-dir\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh97p\" (UniqueName: \"kubernetes.io/projected/a8024a66-c645-46d6-b1c3-f75dd18abb20-kube-api-access-qh97p\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017283    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-systemd\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017310    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-os-release\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-var-lib-openvswitch\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017367    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-run-openvswitch\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.018330 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-log-socket\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-os-release\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-run\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-slash\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017579    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-slash\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017634    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-run-ovn\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-cni-netd\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.017767    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-node-log\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.018323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e-iptables-alerter-script\") pod \"iptables-alerter-99rcs\" (UID: \"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e\") " pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.018488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-etc-openvswitch\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.018542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.019104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.019037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1d3ea62-7e04-4d73-a171-6e573d4e6637-env-overrides\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.019623    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b95fb06-43f5-441b-b46c-60a5c2cc0d9e-konnectivity-ca\") pod \"konnectivity-agent-fl5bn\" (UID: \"0b95fb06-43f5-441b-b46c-60a5c2cc0d9e\") " pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.019696    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-cni-bin\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.019721    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.019926    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-log-socket\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.019965    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-var-lib-openvswitch\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.020006    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-run-ovn-kubernetes\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.020015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-run-openvswitch\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.020049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e-host-slash\") pod \"iptables-alerter-99rcs\" (UID: \"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e\") " pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.020108    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.020133    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-kubelet\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.020173    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-host-run-netns\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.020172    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1d3ea62-7e04-4d73-a171-6e573d4e6637-ovnkube-script-lib\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.020169    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1d3ea62-7e04-4d73-a171-6e573d4e6637-ovnkube-config\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.020266 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.020209    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs podName:6b84e87f-4fbf-471c-bf2f-2c364e4d9a82 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:39.520155937 +0000 UTC m=+2.972777086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs") pod "network-metrics-daemon-flnfk" (UID: "6b84e87f-4fbf-471c-bf2f-2c364e4d9a82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:39.021544 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.019726    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-run-systemd\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.021544 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.020599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1d3ea62-7e04-4d73-a171-6e573d4e6637-systemd-units\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.024091 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.024029    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b95fb06-43f5-441b-b46c-60a5c2cc0d9e-agent-certs\") pod \"konnectivity-agent-fl5bn\" (UID: \"0b95fb06-43f5-441b-b46c-60a5c2cc0d9e\") " pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:27:39.025013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.024994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1d3ea62-7e04-4d73-a171-6e573d4e6637-ovn-node-metrics-cert\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.026855 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.026838    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:39.027000 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.026859    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:39.027000 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.026873    2564 projected.go:196] Error preparing data for projected volume kube-api-access-76bkx for pod openshift-network-diagnostics/network-check-target-8wtkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:39.027000 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.026950    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx podName:42f3ab66-e7e3-4565-8357-475969cf2579 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:39.526934503 +0000 UTC m=+2.979555627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-76bkx" (UniqueName: "kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx") pod "network-check-target-8wtkw" (UID: "42f3ab66-e7e3-4565-8357-475969cf2579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:39.030928 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.030860    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9dw\" (UniqueName: \"kubernetes.io/projected/b1d3ea62-7e04-4d73-a171-6e573d4e6637-kube-api-access-nv9dw\") pod \"ovnkube-node-fm8gt\" (UID: \"b1d3ea62-7e04-4d73-a171-6e573d4e6637\") " pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.031047 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.031018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwj6\" (UniqueName: \"kubernetes.io/projected/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-kube-api-access-xcwj6\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:39.031240 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.031221    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjq8q\" (UniqueName: \"kubernetes.io/projected/a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e-kube-api-access-sjq8q\") pod \"iptables-alerter-99rcs\" (UID: \"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e\") " pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:39.117695 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117667    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c64d459-de22-4e66-a489-4c7ac0a62897-serviceca\") pod \"node-ca-v8rh6\" (UID: \"9c64d459-de22-4e66-a489-4c7ac0a62897\") " pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.117695 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-cni-binary-copy\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-kubernetes\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-device-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117753    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-run-netns\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-lib-modules\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08be5885-b32c-4799-8157-d0c0000bf3ae-tmp\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117828    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-system-cni-dir\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-kubernetes\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-socket-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c64d459-de22-4e66-a489-4c7ac0a62897-host\") pod \"node-ca-v8rh6\" (UID: \"9c64d459-de22-4e66-a489-4c7ac0a62897\") " pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.117926 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs8ms\" (UniqueName: \"kubernetes.io/projected/df9a0647-8600-491a-8042-186017147267-kube-api-access-vs8ms\") pod \"node-resolver-2lkcp\" (UID: \"df9a0647-8600-491a-8042-186017147267\") " pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.117992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-sysctl-conf\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df9a0647-8600-491a-8042-186017147267-hosts-file\") pod \"node-resolver-2lkcp\" (UID: \"df9a0647-8600-491a-8042-186017147267\") " pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-socket-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-var-lib-cni-bin\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118066    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b376d75-6fb9-48ea-8621-e43b18a91e98-kubelet-config\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b376d75-6fb9-48ea-8621-e43b18a91e98-dbus\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118116    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c64d459-de22-4e66-a489-4c7ac0a62897-serviceca\") pod \"node-ca-v8rh6\" (UID: \"9c64d459-de22-4e66-a489-4c7ac0a62897\") " pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-var-lib-kubelet\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b376d75-6fb9-48ea-8621-e43b18a91e98-kubelet-config\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:39.118459 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118336    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-sysctl-conf\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-lib-modules\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-cni-binary-copy\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118611    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-device-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-run-netns\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-var-lib-kubelet\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.118699    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-var-lib-kubelet\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118727    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-tuned\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.118742    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret podName:3b376d75-6fb9-48ea-8621-e43b18a91e98 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:39.618727166 +0000 UTC m=+3.071348310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret") pod "global-pull-secret-syncer-ggttg" (UID: "3b376d75-6fb9-48ea-8621-e43b18a91e98") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8024a66-c645-46d6-b1c3-f75dd18abb20-cni-binary-copy\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118790    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-run-multus-certs\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118807    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118812    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9a0647-8600-491a-8042-186017147267-tmp-dir\") pod \"node-resolver-2lkcp\" (UID: \"df9a0647-8600-491a-8042-186017147267\") " pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-conf-dir\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh97p\" (UniqueName: \"kubernetes.io/projected/a8024a66-c645-46d6-b1c3-f75dd18abb20-kube-api-access-qh97p\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-systemd\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.119112 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-os-release\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.118990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-os-release\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119006    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df9a0647-8600-491a-8042-186017147267-tmp-dir\") pod \"node-resolver-2lkcp\" (UID: \"df9a0647-8600-491a-8042-186017147267\") " pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-run\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-etc-selinux\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119069    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df9a0647-8600-491a-8042-186017147267-hosts-file\") pod \"node-resolver-2lkcp\" (UID: \"df9a0647-8600-491a-8042-186017147267\") " pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-sys-fs\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-modprobe-d\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119121    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-conf-dir\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119075    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-sys\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119170    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-system-cni-dir\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119177    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-sys\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119238    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-etc-kubernetes\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b376d75-6fb9-48ea-8621-e43b18a91e98-dbus\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-os-release\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119284    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-etc-kubernetes\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120015 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119314    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-os-release\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpcr\" (UniqueName: \"kubernetes.io/projected/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-kube-api-access-rgpcr\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-systemd\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-system-cni-dir\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-cni-dir\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-run\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119381    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-var-lib-cni-multus\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-sys-fs\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-hostroot\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-daemon-config\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119447    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-modprobe-d\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119454    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-system-cni-dir\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vklt\" (UniqueName: \"kubernetes.io/projected/a79b2ef9-8af0-49f3-8796-47f1db39e906-kube-api-access-5vklt\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119463    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-etc-selinux\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-var-lib-kubelet\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-cni-dir\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119578    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-var-lib-cni-multus\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.120916 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-hostroot\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119672    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c64d459-de22-4e66-a489-4c7ac0a62897-host\") pod \"node-ca-v8rh6\" (UID: \"9c64d459-de22-4e66-a489-4c7ac0a62897\") " pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119872    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119894    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-run-multus-certs\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119921    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-var-lib-cni-bin\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-cnibin\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-cnibin\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.119987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-run-k8s-cni-cncf-io\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-host\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120032    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5qg2\" (UniqueName: \"kubernetes.io/projected/9c64d459-de22-4e66-a489-4c7ac0a62897-kube-api-access-f5qg2\") pod \"node-ca-v8rh6\" (UID: \"9c64d459-de22-4e66-a489-4c7ac0a62897\") " pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120055    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-socket-dir-parent\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-sysconfig\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-sysctl-d\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lq9v\" (UniqueName: \"kubernetes.io/projected/08be5885-b32c-4799-8157-d0c0000bf3ae-kube-api-access-4lq9v\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120129    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8024a66-c645-46d6-b1c3-f75dd18abb20-cni-binary-copy\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120136    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-daemon-config\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-registration-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120258    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-cnibin\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.121829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120309    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-host\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.122747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120307    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-host-run-k8s-cni-cncf-io\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.122747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-sysconfig\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.122747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a8024a66-c645-46d6-b1c3-f75dd18abb20-multus-socket-dir-parent\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.122747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a79b2ef9-8af0-49f3-8796-47f1db39e906-registration-dir\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.122747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120432    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-sysctl-d\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.122747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.120450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-cnibin\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.122747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.121066    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.122747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.121948    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08be5885-b32c-4799-8157-d0c0000bf3ae-etc-tuned\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.123093 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.123019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal" event={"ID":"d3fecf50f073a1a78674a6ab671580e5","Type":"ContainerStarted","Data":"16e28ed08563bfa8206c25f0fcb694c844137bb2dadab07a5950d387e392dfe3"}
Jul 27 17:27:39.123129 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.123104    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08be5885-b32c-4799-8157-d0c0000bf3ae-tmp\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.124573 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.124539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal" event={"ID":"ebc1b9d3588ffefd5cf60f36ae3035ca","Type":"ContainerStarted","Data":"812573200e857b8bed307c2421d47aaf1ccdd0be1ab78b2a11be796a761c6061"}
Jul 27 17:27:39.132153 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.131935    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpcr\" (UniqueName: \"kubernetes.io/projected/95cbf9d1-0fa8-43af-9900-08b4d8cc7051-kube-api-access-rgpcr\") pod \"multus-additional-cni-plugins-n252c\" (UID: \"95cbf9d1-0fa8-43af-9900-08b4d8cc7051\") " pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.137239 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.137106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs8ms\" (UniqueName: \"kubernetes.io/projected/df9a0647-8600-491a-8042-186017147267-kube-api-access-vs8ms\") pod \"node-resolver-2lkcp\" (UID: \"df9a0647-8600-491a-8042-186017147267\") " pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.137394 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.137372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5qg2\" (UniqueName: \"kubernetes.io/projected/9c64d459-de22-4e66-a489-4c7ac0a62897-kube-api-access-f5qg2\") pod \"node-ca-v8rh6\" (UID: \"9c64d459-de22-4e66-a489-4c7ac0a62897\") " pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.137935 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.137874    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vklt\" (UniqueName: \"kubernetes.io/projected/a79b2ef9-8af0-49f3-8796-47f1db39e906-kube-api-access-5vklt\") pod \"aws-ebs-csi-driver-node-plz8g\" (UID: \"a79b2ef9-8af0-49f3-8796-47f1db39e906\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.138348 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.138101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh97p\" (UniqueName: \"kubernetes.io/projected/a8024a66-c645-46d6-b1c3-f75dd18abb20-kube-api-access-qh97p\") pod \"multus-gqmhw\" (UID: \"a8024a66-c645-46d6-b1c3-f75dd18abb20\") " pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.139324 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.139274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lq9v\" (UniqueName: \"kubernetes.io/projected/08be5885-b32c-4799-8157-d0c0000bf3ae-kube-api-access-4lq9v\") pod \"tuned-26rmm\" (UID: \"08be5885-b32c-4799-8157-d0c0000bf3ae\") " pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.220778 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.220672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:27:39.231882 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.231859    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:27:39.241200 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.241170    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-99rcs"
Jul 27 17:27:39.248174 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.248154    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gqmhw"
Jul 27 17:27:39.257138 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.257111    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g"
Jul 27 17:27:39.263964 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.263944    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26rmm"
Jul 27 17:27:39.273038 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.273011    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2lkcp"
Jul 27 17:27:39.280965 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.280948    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v8rh6"
Jul 27 17:27:39.287546 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.287528    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n252c"
Jul 27 17:27:39.522094 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.522061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:39.522322 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.522219    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:39.522322 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.522286    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs podName:6b84e87f-4fbf-471c-bf2f-2c364e4d9a82 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:40.522265279 +0000 UTC m=+3.974886418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs") pod "network-metrics-daemon-flnfk" (UID: "6b84e87f-4fbf-471c-bf2f-2c364e4d9a82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:39.622886 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.622861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:39.623020 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.622897    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:39.623082 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.623052    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:39.623082 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.623073    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:39.623147 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.623087    2564 projected.go:196] Error preparing data for projected volume kube-api-access-76bkx for pod openshift-network-diagnostics/network-check-target-8wtkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:39.623147 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.623139    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx podName:42f3ab66-e7e3-4565-8357-475969cf2579 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:40.623122231 +0000 UTC m=+4.075743363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-76bkx" (UniqueName: "kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx") pod "network-check-target-8wtkw" (UID: "42f3ab66-e7e3-4565-8357-475969cf2579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:39.623263 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.623140    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:39.623263 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:39.623215    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret podName:3b376d75-6fb9-48ea-8621-e43b18a91e98 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:40.62319583 +0000 UTC m=+4.075816975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret") pod "global-pull-secret-syncer-ggttg" (UID: "3b376d75-6fb9-48ea-8621-e43b18a91e98") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:39.955091 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.954987    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:22:38 +0000 UTC" deadline="2028-01-19 19:16:14.322900076 +0000 UTC"
Jul 27 17:27:39.955091 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:39.955031    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12985h48m34.367871812s"
Jul 27 17:27:40.121172 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:40.121145    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d3ea62_7e04_4d73_a171_6e573d4e6637.slice/crio-aa0da025cd7912c4c7290cb6dc8f48a153258ab24687e4b412c7aa2805ee4d0e WatchSource:0}: Error finding container aa0da025cd7912c4c7290cb6dc8f48a153258ab24687e4b412c7aa2805ee4d0e: Status 404 returned error can't find the container with id aa0da025cd7912c4c7290cb6dc8f48a153258ab24687e4b412c7aa2805ee4d0e
Jul 27 17:27:40.127000 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:40.126976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerStarted","Data":"aa0da025cd7912c4c7290cb6dc8f48a153258ab24687e4b412c7aa2805ee4d0e"}
Jul 27 17:27:40.137547 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:40.137523    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f67c40_5ff8_4ce1_9ed9_7ac390f9ff5e.slice/crio-264ce7d97a127c9c51e15a5f8d31fd6b39e4010e079687a78caa88b188799562 WatchSource:0}: Error finding container 264ce7d97a127c9c51e15a5f8d31fd6b39e4010e079687a78caa88b188799562: Status 404 returned error can't find the container with id 264ce7d97a127c9c51e15a5f8d31fd6b39e4010e079687a78caa88b188799562
Jul 27 17:27:40.138327 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:40.138302    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c64d459_de22_4e66_a489_4c7ac0a62897.slice/crio-d31daad259b62f39a8788dcc348884f11948ae2e8e1c2e1d4c884a8b4139555b WatchSource:0}: Error finding container d31daad259b62f39a8788dcc348884f11948ae2e8e1c2e1d4c884a8b4139555b: Status 404 returned error can't find the container with id d31daad259b62f39a8788dcc348884f11948ae2e8e1c2e1d4c884a8b4139555b
Jul 27 17:27:40.138985 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:40.138878    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9a0647_8600_491a_8042_186017147267.slice/crio-70308950ebaa1d575391c7c9333e608fdca894e7c0a05c134006f957ab7372c6 WatchSource:0}: Error finding container 70308950ebaa1d575391c7c9333e608fdca894e7c0a05c134006f957ab7372c6: Status 404 returned error can't find the container with id 70308950ebaa1d575391c7c9333e608fdca894e7c0a05c134006f957ab7372c6
Jul 27 17:27:40.141613 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:40.141529    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b95fb06_43f5_441b_b46c_60a5c2cc0d9e.slice/crio-30bd5a202b01d61b19eb87d441bfccb02026829421158af934951d30dc579c86 WatchSource:0}: Error finding container 30bd5a202b01d61b19eb87d441bfccb02026829421158af934951d30dc579c86: Status 404 returned error can't find the container with id 30bd5a202b01d61b19eb87d441bfccb02026829421158af934951d30dc579c86
Jul 27 17:27:40.143007 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:40.142983    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cbf9d1_0fa8_43af_9900_08b4d8cc7051.slice/crio-2d80d5c79c3d0ca0c5fdbe1133649c1ede60d5eb7bd08e8735a52ece29081c58 WatchSource:0}: Error finding container 2d80d5c79c3d0ca0c5fdbe1133649c1ede60d5eb7bd08e8735a52ece29081c58: Status 404 returned error can't find the container with id 2d80d5c79c3d0ca0c5fdbe1133649c1ede60d5eb7bd08e8735a52ece29081c58
Jul 27 17:27:40.144152 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:40.144125    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79b2ef9_8af0_49f3_8796_47f1db39e906.slice/crio-cad9898330dd4bbc412b75841af8f72a7031820eae2166c211b2cd221817ce7b WatchSource:0}: Error finding container cad9898330dd4bbc412b75841af8f72a7031820eae2166c211b2cd221817ce7b: Status 404 returned error can't find the container with id cad9898330dd4bbc412b75841af8f72a7031820eae2166c211b2cd221817ce7b
Jul 27 17:27:40.145786 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:40.145765    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8024a66_c645_46d6_b1c3_f75dd18abb20.slice/crio-b622e1b1d508d9c98a09b23ff4e5c834e3fcb409c78cddc6014a8837134b11a1 WatchSource:0}: Error finding container b622e1b1d508d9c98a09b23ff4e5c834e3fcb409c78cddc6014a8837134b11a1: Status 404 returned error can't find the container with id b622e1b1d508d9c98a09b23ff4e5c834e3fcb409c78cddc6014a8837134b11a1
Jul 27 17:27:40.146697 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:27:40.146630    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08be5885_b32c_4799_8157_d0c0000bf3ae.slice/crio-4511b0cf05a828589b74ce9156f8fa9b5922ff2882c97f5226c620faa21d76b8 WatchSource:0}: Error finding container 4511b0cf05a828589b74ce9156f8fa9b5922ff2882c97f5226c620faa21d76b8: Status 404 returned error can't find the container with id 4511b0cf05a828589b74ce9156f8fa9b5922ff2882c97f5226c620faa21d76b8
Jul 27 17:27:40.529121 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:40.528904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:40.529264 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:40.529076    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:40.529334 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:40.529288    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs podName:6b84e87f-4fbf-471c-bf2f-2c364e4d9a82 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:42.529267226 +0000 UTC m=+5.981888365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs") pod "network-metrics-daemon-flnfk" (UID: "6b84e87f-4fbf-471c-bf2f-2c364e4d9a82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:40.631199 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:40.630363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:40.631199 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:40.630467    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:40.631199 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:40.630604    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:40.631199 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:40.630623    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:40.631199 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:40.630635    2564 projected.go:196] Error preparing data for projected volume kube-api-access-76bkx for pod openshift-network-diagnostics/network-check-target-8wtkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:40.631199 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:40.630694    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx podName:42f3ab66-e7e3-4565-8357-475969cf2579 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:42.630675761 +0000 UTC m=+6.083296889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-76bkx" (UniqueName: "kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx") pod "network-check-target-8wtkw" (UID: "42f3ab66-e7e3-4565-8357-475969cf2579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:40.631199 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:40.630773    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:40.631199 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:40.630812    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret podName:3b376d75-6fb9-48ea-8621-e43b18a91e98 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:42.630798731 +0000 UTC m=+6.083419870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret") pod "global-pull-secret-syncer-ggttg" (UID: "3b376d75-6fb9-48ea-8621-e43b18a91e98") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:40.955959 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:40.955875    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:22:38 +0000 UTC" deadline="2028-01-30 02:53:23.210228404 +0000 UTC"
Jul 27 17:27:40.955959 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:40.955907    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13233h25m42.254325386s"
Jul 27 17:27:41.033053 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.033023    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:41.115350 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.115322    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:41.115507 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:41.115462    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:41.115875 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.115855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:41.115967 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:41.115946    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:41.117482 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.117335    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:41.117482 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:41.117440    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:41.140508 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.139685    2564 generic.go:356] "Generic (PLEG): container finished" podID="d3fecf50f073a1a78674a6ab671580e5" containerID="113ba12db54940b6a50a73805bdaf98e28b00171c6dbf9573d06fce2bf845720" exitCode=0
Jul 27 17:27:41.140508 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.139764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal" event={"ID":"d3fecf50f073a1a78674a6ab671580e5","Type":"ContainerDied","Data":"113ba12db54940b6a50a73805bdaf98e28b00171c6dbf9573d06fce2bf845720"}
Jul 27 17:27:41.152887 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.152292    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal" event={"ID":"ebc1b9d3588ffefd5cf60f36ae3035ca","Type":"ContainerStarted","Data":"decf6783f4efdde21bb96a30431b5dec4ce3a360683bdcb8142c935010635b91"}
Jul 27 17:27:41.157557 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.157480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gqmhw" event={"ID":"a8024a66-c645-46d6-b1c3-f75dd18abb20","Type":"ContainerStarted","Data":"b622e1b1d508d9c98a09b23ff4e5c834e3fcb409c78cddc6014a8837134b11a1"}
Jul 27 17:27:41.167937 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.166747    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g" event={"ID":"a79b2ef9-8af0-49f3-8796-47f1db39e906","Type":"ContainerStarted","Data":"cad9898330dd4bbc412b75841af8f72a7031820eae2166c211b2cd221817ce7b"}
Jul 27 17:27:41.168211 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.168140    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v8rh6" event={"ID":"9c64d459-de22-4e66-a489-4c7ac0a62897","Type":"ContainerStarted","Data":"d31daad259b62f39a8788dcc348884f11948ae2e8e1c2e1d4c884a8b4139555b"}
Jul 27 17:27:41.170339 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.170278    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26rmm" event={"ID":"08be5885-b32c-4799-8157-d0c0000bf3ae","Type":"ContainerStarted","Data":"4511b0cf05a828589b74ce9156f8fa9b5922ff2882c97f5226c620faa21d76b8"}
Jul 27 17:27:41.171567 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.171526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n252c" event={"ID":"95cbf9d1-0fa8-43af-9900-08b4d8cc7051","Type":"ContainerStarted","Data":"2d80d5c79c3d0ca0c5fdbe1133649c1ede60d5eb7bd08e8735a52ece29081c58"}
Jul 27 17:27:41.174335 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.174282    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fl5bn" event={"ID":"0b95fb06-43f5-441b-b46c-60a5c2cc0d9e","Type":"ContainerStarted","Data":"30bd5a202b01d61b19eb87d441bfccb02026829421158af934951d30dc579c86"}
Jul 27 17:27:41.184776 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.184561    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2lkcp" event={"ID":"df9a0647-8600-491a-8042-186017147267","Type":"ContainerStarted","Data":"70308950ebaa1d575391c7c9333e608fdca894e7c0a05c134006f957ab7372c6"}
Jul 27 17:27:41.195827 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:41.195800    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-99rcs" event={"ID":"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e","Type":"ContainerStarted","Data":"264ce7d97a127c9c51e15a5f8d31fd6b39e4010e079687a78caa88b188799562"}
Jul 27 17:27:42.222460 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:42.222421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal" event={"ID":"d3fecf50f073a1a78674a6ab671580e5","Type":"ContainerStarted","Data":"5807ac53b334969c0e62c69c871e4b111652d4bb67c516ece896ce7de92dba2b"}
Jul 27 17:27:42.241035 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:42.240966    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-58.ec2.internal" podStartSLOduration=4.240944562 podStartE2EDuration="4.240944562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:42.240829108 +0000 UTC m=+5.693450256" watchObservedRunningTime="2026-07-27 17:27:42.240944562 +0000 UTC m=+5.693565707"
Jul 27 17:27:42.241748 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:42.241706    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-58.ec2.internal" podStartSLOduration=4.241688027 podStartE2EDuration="4.241688027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:41.184737915 +0000 UTC m=+4.637359064" watchObservedRunningTime="2026-07-27 17:27:42.241688027 +0000 UTC m=+5.694309174"
Jul 27 17:27:42.547607 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:42.547543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:42.547777 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:42.547707    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:42.547841 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:42.547778    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs podName:6b84e87f-4fbf-471c-bf2f-2c364e4d9a82 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:46.547756507 +0000 UTC m=+10.000377651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs") pod "network-metrics-daemon-flnfk" (UID: "6b84e87f-4fbf-471c-bf2f-2c364e4d9a82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:42.648691 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:42.647912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:42.648691 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:42.647965    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:42.648691 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:42.648086    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:42.648691 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:42.648150    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret podName:3b376d75-6fb9-48ea-8621-e43b18a91e98 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:46.648129169 +0000 UTC m=+10.100750307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret") pod "global-pull-secret-syncer-ggttg" (UID: "3b376d75-6fb9-48ea-8621-e43b18a91e98") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:42.648691 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:42.648578    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:42.648691 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:42.648595    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:42.648691 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:42.648607    2564 projected.go:196] Error preparing data for projected volume kube-api-access-76bkx for pod openshift-network-diagnostics/network-check-target-8wtkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:42.648691 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:42.648652    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx podName:42f3ab66-e7e3-4565-8357-475969cf2579 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:46.648636727 +0000 UTC m=+10.101257855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-76bkx" (UniqueName: "kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx") pod "network-check-target-8wtkw" (UID: "42f3ab66-e7e3-4565-8357-475969cf2579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:43.120693 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:43.120661    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:43.120854 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:43.120775    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:43.120924 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:43.120899    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:43.121150 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:43.120899    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:43.121150 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:43.120991    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:43.121150 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:43.121106    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:45.114422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:45.114345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:45.114857 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:45.114468    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:45.114857 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:45.114522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:45.114857 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:45.114639    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:45.114857 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:45.114699    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:45.114857 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:45.114767    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:46.581364 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:46.581329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:46.581835 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:46.581480    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:46.581835 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:46.581560    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs podName:6b84e87f-4fbf-471c-bf2f-2c364e4d9a82 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:54.581538853 +0000 UTC m=+18.034159976 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs") pod "network-metrics-daemon-flnfk" (UID: "6b84e87f-4fbf-471c-bf2f-2c364e4d9a82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:46.682934 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:46.682893    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:46.683105 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:46.683081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:46.683310 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:46.683292    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:46.683396 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:46.683384    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret podName:3b376d75-6fb9-48ea-8621-e43b18a91e98 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:54.683365119 +0000 UTC m=+18.135986286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret") pod "global-pull-secret-syncer-ggttg" (UID: "3b376d75-6fb9-48ea-8621-e43b18a91e98") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:46.687296 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:46.683746    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:46.687296 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:46.683775    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:46.687296 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:46.683798    2564 projected.go:196] Error preparing data for projected volume kube-api-access-76bkx for pod openshift-network-diagnostics/network-check-target-8wtkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:46.687296 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:46.683854    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx podName:42f3ab66-e7e3-4565-8357-475969cf2579 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:54.683832539 +0000 UTC m=+18.136453680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-76bkx" (UniqueName: "kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx") pod "network-check-target-8wtkw" (UID: "42f3ab66-e7e3-4565-8357-475969cf2579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:47.117937 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:47.117906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:47.118109 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:47.118017    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:47.118363 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:47.118342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:47.118499 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:47.118452    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:47.118577 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:47.118509    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:47.118641 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:47.118575    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:49.117833 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:49.117783    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:49.118379 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:49.117904    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:49.118379 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:49.117909    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:49.118379 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:49.117952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:49.118379 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:49.118054    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:49.118379 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:49.118125    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:51.117068 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:51.117038    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:51.117068 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:51.117062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:51.117586 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:51.117153    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:51.117586 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:51.117154    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:51.117586 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:51.117273    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:51.117586 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:51.117347    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:53.117738 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:53.117708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:53.118161 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:53.117708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:53.118161 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:53.117824    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:53.118161 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:53.117708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:53.118161 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:53.117913    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:53.118161 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:53.118014    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:54.646334 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:54.646293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:54.646828 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:54.646468    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:54.646828 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:54.646549    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs podName:6b84e87f-4fbf-471c-bf2f-2c364e4d9a82 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:10.646528527 +0000 UTC m=+34.099149670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs") pod "network-metrics-daemon-flnfk" (UID: "6b84e87f-4fbf-471c-bf2f-2c364e4d9a82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:54.746754 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:54.746720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:54.746754 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:54.746758    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:54.746941 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:54.746872    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:54.746941 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:54.746889    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:54.746941 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:54.746906    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:54.746941 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:54.746919    2564 projected.go:196] Error preparing data for projected volume kube-api-access-76bkx for pod openshift-network-diagnostics/network-check-target-8wtkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:54.746941 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:54.746939    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret podName:3b376d75-6fb9-48ea-8621-e43b18a91e98 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:10.746920923 +0000 UTC m=+34.199542062 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret") pod "global-pull-secret-syncer-ggttg" (UID: "3b376d75-6fb9-48ea-8621-e43b18a91e98") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:54.747090 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:54.746958    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx podName:42f3ab66-e7e3-4565-8357-475969cf2579 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:10.746946336 +0000 UTC m=+34.199567480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-76bkx" (UniqueName: "kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx") pod "network-check-target-8wtkw" (UID: "42f3ab66-e7e3-4565-8357-475969cf2579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:55.117922 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:55.117892    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:55.118114 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:55.117936    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:55.118114 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:55.118014    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:55.118114 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:55.118026    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:55.118301 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:55.118114    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:55.118301 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:55.118211    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:57.116040 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:57.116003    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:57.116040 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:57.116042    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:57.116567 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:57.116012    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:57.116567 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:57.116120    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:57.116567 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:57.116233    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:57.116567 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:57.116314    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:59.114310 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.114132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:27:59.114658 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.114132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:27:59.114658 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:59.114381    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:27:59.114658 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.114156    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:27:59.114658 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:59.114508    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:27:59.114658 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:27:59.114542    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:27:59.258074 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.258045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v8rh6" event={"ID":"9c64d459-de22-4e66-a489-4c7ac0a62897","Type":"ContainerStarted","Data":"ba108e09015c3510b23e774660776f9a198f94e7f328d91203fbaeb8bff641ef"}
Jul 27 17:27:59.259719 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.259700    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:27:59.260021 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.260001    2564 generic.go:356] "Generic (PLEG): container finished" podID="b1d3ea62-7e04-4d73-a171-6e573d4e6637" containerID="65fa352ee6967bf243edfe4c33ece225a5cb5f5ea4ec523b6fadbf29a0f32b68" exitCode=1
Jul 27 17:27:59.260087 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.260063    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerDied","Data":"65fa352ee6967bf243edfe4c33ece225a5cb5f5ea4ec523b6fadbf29a0f32b68"}
Jul 27 17:27:59.260146 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.260131    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerStarted","Data":"99d07726ffb905eaff1dc9700d822424ea4d5afe0de02eb5d54bc2cfc191b02e"}
Jul 27 17:27:59.261379 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.261359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26rmm" event={"ID":"08be5885-b32c-4799-8157-d0c0000bf3ae","Type":"ContainerStarted","Data":"137d27f7ba963dedb64a1e48b2e9d59b05d712e8b55e1118f3691b26d49cedad"}
Jul 27 17:27:59.262892 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.262870    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n252c" event={"ID":"95cbf9d1-0fa8-43af-9900-08b4d8cc7051","Type":"ContainerStarted","Data":"38735c2d030bca3ecfc677628a914570fee2fa1831b28b2a68dcf01f4c095f6b"}
Jul 27 17:27:59.264923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.264900    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fl5bn" event={"ID":"0b95fb06-43f5-441b-b46c-60a5c2cc0d9e","Type":"ContainerStarted","Data":"e810bfef3e99901f5fdfdef0da1843d390876964301eef3a8b23249e06d9b934"}
Jul 27 17:27:59.266416 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.266387    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2lkcp" event={"ID":"df9a0647-8600-491a-8042-186017147267","Type":"ContainerStarted","Data":"81c425771c6657ef692edebfe2d89aed9c1c9967383270b39c38d48187d9d611"}
Jul 27 17:27:59.267993 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.267974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g" event={"ID":"a79b2ef9-8af0-49f3-8796-47f1db39e906","Type":"ContainerStarted","Data":"56ac3e1ec88b3fc591abcac0eaf13b95c499374ff0e0f60c0b6fc0dc655be8d8"}
Jul 27 17:27:59.274172 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.274137    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v8rh6" podStartSLOduration=3.582225178 podStartE2EDuration="22.274126961s" totalImagesPullingTime="18.691901783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:59.273970966 +0000 UTC m=+22.726592113" watchObservedRunningTime="2026-07-27 17:27:59.274126961 +0000 UTC m=+22.726748107"
Jul 27 17:27:59.291929 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.291891    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-26rmm" podStartSLOduration=3.559620636 podStartE2EDuration="22.291880258s" totalImagesPullingTime="18.732259622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:59.291526522 +0000 UTC m=+22.744147681" watchObservedRunningTime="2026-07-27 17:27:59.291880258 +0000 UTC m=+22.744501401"
Jul 27 17:27:59.308221 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.308170    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2lkcp" podStartSLOduration=3.565973645 podStartE2EDuration="22.308158004s" totalImagesPullingTime="18.742184359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:59.308064356 +0000 UTC m=+22.760685498" watchObservedRunningTime="2026-07-27 17:27:59.308158004 +0000 UTC m=+22.760779151"
Jul 27 17:27:59.361556 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:27:59.361367    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fl5bn" podStartSLOduration=4.075318995 podStartE2EDuration="22.36134863s" totalImagesPullingTime="18.286029635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:59.361100249 +0000 UTC m=+22.813721397" watchObservedRunningTime="2026-07-27 17:27:59.36134863 +0000 UTC m=+22.813969779"
Jul 27 17:28:00.272047 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.272007    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gqmhw" event={"ID":"a8024a66-c645-46d6-b1c3-f75dd18abb20","Type":"ContainerStarted","Data":"8c845c78965595fc40f09d2c0bd0901c87838ecb5faf7c82d9c5709ec87dbd0d"}
Jul 27 17:28:00.275599 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.275580    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:28:00.276211 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.276164    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerStarted","Data":"3d396f160f2e6c2d8a63704353b1863ece9a78ca6daf0f9e87ca7d0fc7df6c6f"}
Jul 27 17:28:00.276302 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.276252    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerStarted","Data":"8efae0650f804f5c6183688d018ed199b01d745707bd4dd02d33ae36ec9d2386"}
Jul 27 17:28:00.276302 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.276292    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerStarted","Data":"7528b886be53b0ea51ba222c2fdf0540cf2fe10f4405cf8fbc8f488101da685e"}
Jul 27 17:28:00.276422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.276334    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerStarted","Data":"bc68bed92d8f4fdef7ef1aa5670047a9022dae37468f60932b164ea690847582"}
Jul 27 17:28:00.278065 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.278042    2564 generic.go:356] "Generic (PLEG): container finished" podID="95cbf9d1-0fa8-43af-9900-08b4d8cc7051" containerID="38735c2d030bca3ecfc677628a914570fee2fa1831b28b2a68dcf01f4c095f6b" exitCode=0
Jul 27 17:28:00.278152 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.278110    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n252c" event={"ID":"95cbf9d1-0fa8-43af-9900-08b4d8cc7051","Type":"ContainerDied","Data":"38735c2d030bca3ecfc677628a914570fee2fa1831b28b2a68dcf01f4c095f6b"}
Jul 27 17:28:00.280365 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.280337    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-99rcs" event={"ID":"a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e","Type":"ContainerStarted","Data":"9ec94d260fc9a87987e63f48884d032f695992a224fe517f6ab340a1f98134a2"}
Jul 27 17:28:00.292709 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.292670    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gqmhw" podStartSLOduration=4.046909046 podStartE2EDuration="23.292658244s" totalImagesPullingTime="19.245749198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:00.29104847 +0000 UTC m=+23.743669617" watchObservedRunningTime="2026-07-27 17:28:00.292658244 +0000 UTC m=+23.745279390"
Jul 27 17:28:00.307619 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:00.307576    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-99rcs" podStartSLOduration=4.588071855 podStartE2EDuration="23.307563582s" totalImagesPullingTime="18.719491727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:00.307151699 +0000 UTC m=+23.759772846" watchObservedRunningTime="2026-07-27 17:28:00.307563582 +0000 UTC m=+23.760184728"
Jul 27 17:28:01.003493 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.003467    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:28:01.004010 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.003991    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:28:01.114964 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.114936    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:01.115121 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.114939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:28:01.115121 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:01.115046    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:28:01.115121 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.115071    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:01.115312 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:01.115218    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:28:01.115312 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:01.115276    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:28:01.142476 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.142454    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 17:28:01.284977 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.284896    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g" event={"ID":"a79b2ef9-8af0-49f3-8796-47f1db39e906","Type":"ContainerStarted","Data":"6cd8526cb9007cb69d9da8ae6f1d7a5e720ba24dd27313605f4679d3e7274d46"}
Jul 27 17:28:01.285473 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.285283    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:28:01.285744 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.285719    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fl5bn"
Jul 27 17:28:01.974557 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.974464    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T17:28:01.142472688Z","UUID":"34939d47-b700-4e5c-b8b2-645f14f07513","Handler":null,"Name":"","Endpoint":""}
Jul 27 17:28:01.976012 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.975989    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 17:28:01.976131 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:01.976031    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 17:28:02.291177 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:02.291104    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:28:02.291653 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:02.291620    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerStarted","Data":"e06dd279944de65f7dadb8d4a156c6b912f0d174c2e3a9385bd0b11cc8f033b0"}
Jul 27 17:28:03.115329 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:03.115122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:28:03.115573 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:03.115439    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:28:03.115573 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:03.115149    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:03.115573 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:03.115122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:03.115573 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:03.115539    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:28:03.115777 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:03.115601    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:28:04.297787 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:04.297581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g" event={"ID":"a79b2ef9-8af0-49f3-8796-47f1db39e906","Type":"ContainerStarted","Data":"bf429f240e878af0f32301f6becca7198188120b0086d45f2bdd681b156e81a0"}
Jul 27 17:28:04.300964 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:04.300941    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:28:04.301380 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:04.301357    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerStarted","Data":"9ccf804ec0b9e1f05a66ee61e1a7ea10a5650b338d777536165efa23ebb71c0b"}
Jul 27 17:28:04.301625 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:04.301607    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:28:04.301759 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:04.301743    2564 scope.go:117] "RemoveContainer" containerID="65fa352ee6967bf243edfe4c33ece225a5cb5f5ea4ec523b6fadbf29a0f32b68"
Jul 27 17:28:04.303227 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:04.303205    2564 generic.go:356] "Generic (PLEG): container finished" podID="95cbf9d1-0fa8-43af-9900-08b4d8cc7051" containerID="7a54aa4879a59384ae5f2134da4a685d8e5388ffdb53130b3249b12b23dee7c8" exitCode=0
Jul 27 17:28:04.303322 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:04.303240    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n252c" event={"ID":"95cbf9d1-0fa8-43af-9900-08b4d8cc7051","Type":"ContainerDied","Data":"7a54aa4879a59384ae5f2134da4a685d8e5388ffdb53130b3249b12b23dee7c8"}
Jul 27 17:28:04.317146 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:04.317126    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:28:04.317753 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:04.317720    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plz8g" podStartSLOduration=4.356946283 podStartE2EDuration="27.317707183s" totalImagesPullingTime="22.9607609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:04.316984974 +0000 UTC m=+27.769606124" watchObservedRunningTime="2026-07-27 17:28:04.317707183 +0000 UTC m=+27.770328328"
Jul 27 17:28:05.118069 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:05.117997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:05.118280 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:05.118111    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:05.118280 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:05.118116    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:28:05.118280 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:05.118174    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:28:05.118421 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:05.118299    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:28:05.118464 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:05.118411    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:28:05.308518 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:05.308489    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:28:05.308977 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:05.308928    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" event={"ID":"b1d3ea62-7e04-4d73-a171-6e573d4e6637","Type":"ContainerStarted","Data":"bbdda8858d7c6bf01af9b6986ef2009d0b17be4fe037fed7bd597a24bc5bb8c8"}
Jul 27 17:28:05.309211 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:05.309173    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:28:05.309385 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:05.309370    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:28:05.323971 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:05.323948    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:28:05.341785 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:05.341749    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt" podStartSLOduration=10.565559689 podStartE2EDuration="29.341736869s" totalImagesPullingTime="18.77617718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:05.34037308 +0000 UTC m=+28.792994237" watchObservedRunningTime="2026-07-27 17:28:05.341736869 +0000 UTC m=+28.794358014"
Jul 27 17:28:06.133382 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:06.133346    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8wtkw"]
Jul 27 17:28:06.133559 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:06.133522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:06.133654 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:06.133625    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:28:06.136154 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:06.136129    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ggttg"]
Jul 27 17:28:06.136302 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:06.136227    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:06.136348 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:06.136297    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:28:06.150888 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:06.150858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flnfk"]
Jul 27 17:28:06.151004 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:06.150956    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:28:06.151064 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:06.151046    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:28:06.313747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:06.313705    2564 generic.go:356] "Generic (PLEG): container finished" podID="95cbf9d1-0fa8-43af-9900-08b4d8cc7051" containerID="57c087d4f6cca3d393618e6dd776a6a66cbc683b1d8c8523e15093b7aaf1142d" exitCode=0
Jul 27 17:28:06.314171 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:06.313796    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n252c" event={"ID":"95cbf9d1-0fa8-43af-9900-08b4d8cc7051","Type":"ContainerDied","Data":"57c087d4f6cca3d393618e6dd776a6a66cbc683b1d8c8523e15093b7aaf1142d"}
Jul 27 17:28:08.114509 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:08.114341    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:08.115041 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:08.114338    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:28:08.115041 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:08.114602    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:28:08.115041 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:08.114397    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:08.115041 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:08.114665    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:28:08.115041 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:08.114722    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:28:08.319923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:08.319885    2564 generic.go:356] "Generic (PLEG): container finished" podID="95cbf9d1-0fa8-43af-9900-08b4d8cc7051" containerID="78b5d779fb7bc9e8f94f0e6eb907e61e59ee8eec3eece05a96d4a41aaaa8d24b" exitCode=0
Jul 27 17:28:08.320054 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:08.319951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n252c" event={"ID":"95cbf9d1-0fa8-43af-9900-08b4d8cc7051","Type":"ContainerDied","Data":"78b5d779fb7bc9e8f94f0e6eb907e61e59ee8eec3eece05a96d4a41aaaa8d24b"}
Jul 27 17:28:10.115057 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.115017    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:10.115654 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.115017    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:28:10.115654 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.115165    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8wtkw" podUID="42f3ab66-e7e3-4565-8357-475969cf2579"
Jul 27 17:28:10.115654 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.115017    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:10.115654 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.115251    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flnfk" podUID="6b84e87f-4fbf-471c-bf2f-2c364e4d9a82"
Jul 27 17:28:10.115654 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.115336    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggttg" podUID="3b376d75-6fb9-48ea-8621-e43b18a91e98"
Jul 27 17:28:10.672789 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.672756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:28:10.672946 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.672892    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:10.672946 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.672940    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs podName:6b84e87f-4fbf-471c-bf2f-2c364e4d9a82 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.672926343 +0000 UTC m=+66.125547467 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs") pod "network-metrics-daemon-flnfk" (UID: "6b84e87f-4fbf-471c-bf2f-2c364e4d9a82") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:10.773487 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.773458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:10.773617 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.773501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:10.773689 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.773644    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:10.773689 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.773660    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:10.773689 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.773687    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:10.773822 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.773707    2564 projected.go:196] Error preparing data for projected volume kube-api-access-76bkx for pod openshift-network-diagnostics/network-check-target-8wtkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:10.773822 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.773717    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret podName:3b376d75-6fb9-48ea-8621-e43b18a91e98 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.773694923 +0000 UTC m=+66.226316061 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret") pod "global-pull-secret-syncer-ggttg" (UID: "3b376d75-6fb9-48ea-8621-e43b18a91e98") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:10.773822 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:10.773765    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx podName:42f3ab66-e7e3-4565-8357-475969cf2579 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.773744364 +0000 UTC m=+66.226365509 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-76bkx" (UniqueName: "kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx") pod "network-check-target-8wtkw" (UID: "42f3ab66-e7e3-4565-8357-475969cf2579") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:10.856629 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.856599    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-58.ec2.internal" event="NodeReady"
Jul 27 17:28:10.856804 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.856759    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 17:28:10.892886 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.892854    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-666b8c6bcb-f2nbx"]
Jul 27 17:28:10.906793 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.906762    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xt257"]
Jul 27 17:28:10.906963 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.906900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:10.910701 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.910557    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 17:28:10.910701 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.910579    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h7n9r\""
Jul 27 17:28:10.910701 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.910649    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 17:28:10.910701 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.910654    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 17:28:10.917082 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.917062    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 17:28:10.920399 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.920380    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666b8c6bcb-f2nbx"]
Jul 27 17:28:10.920508 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.920421    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mjj85"]
Jul 27 17:28:10.920508 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.920501    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:10.923321 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.923273    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-swd8n\""
Jul 27 17:28:10.923321 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.923296    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 17:28:10.923476 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.923391    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 17:28:10.935645 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.935627    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xt257"]
Jul 27 17:28:10.935744 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.935714    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mjj85"]
Jul 27 17:28:10.935808 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.935784    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mjj85"
Jul 27 17:28:10.939031 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.938899    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z94n7\""
Jul 27 17:28:10.939031 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.938928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 17:28:10.939031 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.938971    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 17:28:10.939031 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:10.938991    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 17:28:11.075519 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075479    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-registry-certificates\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.075705 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075531    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-bound-sa-token\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.075705 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27wn\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-kube-api-access-c27wn\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.075705 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332d404e-0e71-485a-8a0d-05951cf4752d-config-volume\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.075705 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac-cert\") pod \"ingress-canary-mjj85\" (UID: \"7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac\") " pod="openshift-ingress-canary/ingress-canary-mjj85"
Jul 27 17:28:11.075705 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-installation-pull-secrets\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.075963 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075753    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-trusted-ca\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.075963 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075779    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332d404e-0e71-485a-8a0d-05951cf4752d-tmp-dir\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.075963 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075801    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqv4d\" (UniqueName: \"kubernetes.io/projected/7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac-kube-api-access-xqv4d\") pod \"ingress-canary-mjj85\" (UID: \"7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac\") " pod="openshift-ingress-canary/ingress-canary-mjj85"
Jul 27 17:28:11.075963 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075835    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abb877f-c36e-4524-bf95-241c606335ce-ca-trust-extracted\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.075963 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075851    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gw2s\" (UniqueName: \"kubernetes.io/projected/332d404e-0e71-485a-8a0d-05951cf4752d-kube-api-access-7gw2s\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.075963 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075915    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.075963 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-image-registry-private-configuration\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.076334 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.075971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.176724 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.176646    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-trusted-ca\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.176724 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.176680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332d404e-0e71-485a-8a0d-05951cf4752d-tmp-dir\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.176724 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.176698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqv4d\" (UniqueName: \"kubernetes.io/projected/7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac-kube-api-access-xqv4d\") pod \"ingress-canary-mjj85\" (UID: \"7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac\") " pod="openshift-ingress-canary/ingress-canary-mjj85"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.176734    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abb877f-c36e-4524-bf95-241c606335ce-ca-trust-extracted\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.176899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gw2s\" (UniqueName: \"kubernetes.io/projected/332d404e-0e71-485a-8a0d-05951cf4752d-kube-api-access-7gw2s\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.176970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177006    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-image-registry-private-configuration\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-registry-certificates\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-bound-sa-token\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27wn\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-kube-api-access-c27wn\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332d404e-0e71-485a-8a0d-05951cf4752d-config-volume\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177209    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac-cert\") pod \"ingress-canary-mjj85\" (UID: \"7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac\") " pod="openshift-ingress-canary/ingress-canary-mjj85"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177243    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-installation-pull-secrets\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177332    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332d404e-0e71-485a-8a0d-05951cf4752d-tmp-dir\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.177422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177379    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abb877f-c36e-4524-bf95-241c606335ce-ca-trust-extracted\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177989 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.177455    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:28:11.177989 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.177470    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666b8c6bcb-f2nbx: secret "image-registry-tls" not found
Jul 27 17:28:11.177989 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.177524    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls podName:9abb877f-c36e-4524-bf95-241c606335ce nodeName:}" failed. No retries permitted until 2026-07-27 17:28:11.677504606 +0000 UTC m=+35.130125744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls") pod "image-registry-666b8c6bcb-f2nbx" (UID: "9abb877f-c36e-4524-bf95-241c606335ce") : secret "image-registry-tls" not found
Jul 27 17:28:11.177989 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.177783    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:11.177989 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177856    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-registry-certificates\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177989 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.177872    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls podName:332d404e-0e71-485a-8a0d-05951cf4752d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:11.677855006 +0000 UTC m=+35.130476147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls") pod "dns-default-xt257" (UID: "332d404e-0e71-485a-8a0d-05951cf4752d") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:11.177989 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177939    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-trusted-ca\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.177989 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.177941    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332d404e-0e71-485a-8a0d-05951cf4752d-config-volume\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.181778 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.181754    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-installation-pull-secrets\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.181778 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.181770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac-cert\") pod \"ingress-canary-mjj85\" (UID: \"7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac\") " pod="openshift-ingress-canary/ingress-canary-mjj85"
Jul 27 17:28:11.181949 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.181761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-image-registry-private-configuration\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.192748 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.192724    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gw2s\" (UniqueName: \"kubernetes.io/projected/332d404e-0e71-485a-8a0d-05951cf4752d-kube-api-access-7gw2s\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.193402 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.193353    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqv4d\" (UniqueName: \"kubernetes.io/projected/7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac-kube-api-access-xqv4d\") pod \"ingress-canary-mjj85\" (UID: \"7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac\") " pod="openshift-ingress-canary/ingress-canary-mjj85"
Jul 27 17:28:11.193814 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.193797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-bound-sa-token\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.195159 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.195141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27wn\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-kube-api-access-c27wn\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.245488 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.245455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mjj85"
Jul 27 17:28:11.682009 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.681964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:11.682199 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:11.682024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:11.682199 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.682140    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:11.682345 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.682210    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:28:11.682345 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.682225    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls podName:332d404e-0e71-485a-8a0d-05951cf4752d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:12.682203838 +0000 UTC m=+36.134824974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls") pod "dns-default-xt257" (UID: "332d404e-0e71-485a-8a0d-05951cf4752d") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:11.682345 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.682228    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666b8c6bcb-f2nbx: secret "image-registry-tls" not found
Jul 27 17:28:11.682345 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:11.682281    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls podName:9abb877f-c36e-4524-bf95-241c606335ce nodeName:}" failed. No retries permitted until 2026-07-27 17:28:12.682264105 +0000 UTC m=+36.134885231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls") pod "image-registry-666b8c6bcb-f2nbx" (UID: "9abb877f-c36e-4524-bf95-241c606335ce") : secret "image-registry-tls" not found
Jul 27 17:28:12.114348 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.114309    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:12.114547 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.114309    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:12.114615 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.114313    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:28:12.118739 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.118713    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:28:12.118739 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.118723    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:28:12.118917 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.118739    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jhb7r\""
Jul 27 17:28:12.118917 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.118787    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:28:12.118917 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.118827    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p44n4\""
Jul 27 17:28:12.118917 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.118746    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:28:12.478227 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.478195    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mjj85"]
Jul 27 17:28:12.532283 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:28:12.532252    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd586f7_6fb6_4d6d_a0fc_2b633a11f5ac.slice/crio-d016d6f290bb3958a6be35dacb74876d05bd8010520eef18f2583bb832307d37 WatchSource:0}: Error finding container d016d6f290bb3958a6be35dacb74876d05bd8010520eef18f2583bb832307d37: Status 404 returned error can't find the container with id d016d6f290bb3958a6be35dacb74876d05bd8010520eef18f2583bb832307d37
Jul 27 17:28:12.692283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.692192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:12.692283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:12.692248    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:12.692515 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:12.692389    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:12.692515 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:12.692409    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:28:12.692515 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:12.692431    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666b8c6bcb-f2nbx: secret "image-registry-tls" not found
Jul 27 17:28:12.692515 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:12.692467    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls podName:332d404e-0e71-485a-8a0d-05951cf4752d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:14.692450822 +0000 UTC m=+38.145071947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls") pod "dns-default-xt257" (UID: "332d404e-0e71-485a-8a0d-05951cf4752d") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:12.692515 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:12.692494    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls podName:9abb877f-c36e-4524-bf95-241c606335ce nodeName:}" failed. No retries permitted until 2026-07-27 17:28:14.692477625 +0000 UTC m=+38.145098773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls") pod "image-registry-666b8c6bcb-f2nbx" (UID: "9abb877f-c36e-4524-bf95-241c606335ce") : secret "image-registry-tls" not found
Jul 27 17:28:13.334232 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:13.334193    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mjj85" event={"ID":"7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac","Type":"ContainerStarted","Data":"d016d6f290bb3958a6be35dacb74876d05bd8010520eef18f2583bb832307d37"}
Jul 27 17:28:13.337379 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:13.337350    2564 generic.go:356] "Generic (PLEG): container finished" podID="95cbf9d1-0fa8-43af-9900-08b4d8cc7051" containerID="649381ea48ae74ca8df3dfbc9b6a433a507754df4eb6aa11b781c3066b467629" exitCode=0
Jul 27 17:28:13.337496 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:13.337407    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n252c" event={"ID":"95cbf9d1-0fa8-43af-9900-08b4d8cc7051","Type":"ContainerDied","Data":"649381ea48ae74ca8df3dfbc9b6a433a507754df4eb6aa11b781c3066b467629"}
Jul 27 17:28:14.343578 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:14.343540    2564 generic.go:356] "Generic (PLEG): container finished" podID="95cbf9d1-0fa8-43af-9900-08b4d8cc7051" containerID="fc79758336af2a37f9918b65438d9ecbdc01cbc7eeb96ec4343a8d684873919a" exitCode=0
Jul 27 17:28:14.344085 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:14.343593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n252c" event={"ID":"95cbf9d1-0fa8-43af-9900-08b4d8cc7051","Type":"ContainerDied","Data":"fc79758336af2a37f9918b65438d9ecbdc01cbc7eeb96ec4343a8d684873919a"}
Jul 27 17:28:14.709668 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:14.709498    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:14.709771 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:14.709709    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:14.709771 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:14.709649    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:14.709911 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:14.709824    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls podName:332d404e-0e71-485a-8a0d-05951cf4752d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:18.709801004 +0000 UTC m=+42.162422145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls") pod "dns-default-xt257" (UID: "332d404e-0e71-485a-8a0d-05951cf4752d") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:14.709911 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:14.709824    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:28:14.709911 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:14.709846    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666b8c6bcb-f2nbx: secret "image-registry-tls" not found
Jul 27 17:28:14.709911 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:14.709884    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls podName:9abb877f-c36e-4524-bf95-241c606335ce nodeName:}" failed. No retries permitted until 2026-07-27 17:28:18.709873343 +0000 UTC m=+42.162494481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls") pod "image-registry-666b8c6bcb-f2nbx" (UID: "9abb877f-c36e-4524-bf95-241c606335ce") : secret "image-registry-tls" not found
Jul 27 17:28:15.346980 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:15.346939    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mjj85" event={"ID":"7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac","Type":"ContainerStarted","Data":"303587815c2d4cbaf84d5283617ac2f825ed9d01aba21a65d7a4187155eb12fc"}
Jul 27 17:28:15.349986 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:15.349956    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n252c" event={"ID":"95cbf9d1-0fa8-43af-9900-08b4d8cc7051","Type":"ContainerStarted","Data":"a3545c81696f565afc1e25b8ef455492cdb14ed89b8d1d01dde287febeed8b56"}
Jul 27 17:28:15.374407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:15.374359    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mjj85" podStartSLOduration=3.2672394479999998 podStartE2EDuration="5.374343144s" totalImagesPullingTime="2.107103696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:15.373207166 +0000 UTC m=+38.825828309" watchObservedRunningTime="2026-07-27 17:28:15.374343144 +0000 UTC m=+38.826964292"
Jul 27 17:28:15.408488 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:15.408447    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n252c" podStartSLOduration=9.100560715 podStartE2EDuration="38.408433292s" totalImagesPullingTime="29.307872577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:15.406006136 +0000 UTC m=+38.858627282" watchObservedRunningTime="2026-07-27 17:28:15.408433292 +0000 UTC m=+38.861054434"
Jul 27 17:28:18.737642 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:18.737605    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:18.737642 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:18.737647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:18.738107 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:18.737754    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:28:18.738107 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:18.737756    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:18.738107 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:18.737817    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls podName:332d404e-0e71-485a-8a0d-05951cf4752d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:26.737802829 +0000 UTC m=+50.190423954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls") pod "dns-default-xt257" (UID: "332d404e-0e71-485a-8a0d-05951cf4752d") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:18.738107 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:18.737765    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666b8c6bcb-f2nbx: secret "image-registry-tls" not found
Jul 27 17:28:18.738107 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:18.737897    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls podName:9abb877f-c36e-4524-bf95-241c606335ce nodeName:}" failed. No retries permitted until 2026-07-27 17:28:26.737885093 +0000 UTC m=+50.190506228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls") pod "image-registry-666b8c6bcb-f2nbx" (UID: "9abb877f-c36e-4524-bf95-241c606335ce") : secret "image-registry-tls" not found
Jul 27 17:28:26.794692 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:26.794655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:26.795197 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:26.794701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:26.795197 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:26.794811    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:26.795197 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:26.794848    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:28:26.795197 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:26.794862    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666b8c6bcb-f2nbx: secret "image-registry-tls" not found
Jul 27 17:28:26.795197 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:26.794890    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls podName:332d404e-0e71-485a-8a0d-05951cf4752d nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.794873381 +0000 UTC m=+66.247494505 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls") pod "dns-default-xt257" (UID: "332d404e-0e71-485a-8a0d-05951cf4752d") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:26.795197 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:26.794913    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls podName:9abb877f-c36e-4524-bf95-241c606335ce nodeName:}" failed. No retries permitted until 2026-07-27 17:28:42.79489704 +0000 UTC m=+66.247518183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls") pod "image-registry-666b8c6bcb-f2nbx" (UID: "9abb877f-c36e-4524-bf95-241c606335ce") : secret "image-registry-tls" not found
Jul 27 17:28:37.331358 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:37.331332    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fm8gt"
Jul 27 17:28:42.698034 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.697983    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:28:42.702131 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.702113    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:28:42.709136 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:42.709119    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:28:42.709212 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:42.709202    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs podName:6b84e87f-4fbf-471c-bf2f-2c364e4d9a82 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:46.709173259 +0000 UTC m=+130.161794396 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs") pod "network-metrics-daemon-flnfk" (UID: "6b84e87f-4fbf-471c-bf2f-2c364e4d9a82") : secret "metrics-daemon-secret" not found
Jul 27 17:28:42.799116 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.799077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:28:42.799116 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.799122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:28:42.799404 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.799156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:42.799404 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.799177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:42.799404 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:42.799256    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:28:42.799404 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:42.799266    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:42.799404 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:42.799275    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666b8c6bcb-f2nbx: secret "image-registry-tls" not found
Jul 27 17:28:42.799404 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:42.799343    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls podName:9abb877f-c36e-4524-bf95-241c606335ce nodeName:}" failed. No retries permitted until 2026-07-27 17:29:14.799323248 +0000 UTC m=+98.251944386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls") pod "image-registry-666b8c6bcb-f2nbx" (UID: "9abb877f-c36e-4524-bf95-241c606335ce") : secret "image-registry-tls" not found
Jul 27 17:28:42.799404 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:28:42.799358    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls podName:332d404e-0e71-485a-8a0d-05951cf4752d nodeName:}" failed. No retries permitted until 2026-07-27 17:29:14.799351075 +0000 UTC m=+98.251972198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls") pod "dns-default-xt257" (UID: "332d404e-0e71-485a-8a0d-05951cf4752d") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:42.801887 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.801870    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:28:42.801952 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.801927    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:28:42.813026 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.812994    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:28:42.814018 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.813995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b376d75-6fb9-48ea-8621-e43b18a91e98-original-pull-secret\") pod \"global-pull-secret-syncer-ggttg\" (UID: \"3b376d75-6fb9-48ea-8621-e43b18a91e98\") " pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:42.822568 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:42.822544    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bkx\" (UniqueName: \"kubernetes.io/projected/42f3ab66-e7e3-4565-8357-475969cf2579-kube-api-access-76bkx\") pod \"network-check-target-8wtkw\" (UID: \"42f3ab66-e7e3-4565-8357-475969cf2579\") " pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:43.028821 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:43.028788    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggttg"
Jul 27 17:28:43.039873 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:43.039846    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jhb7r\""
Jul 27 17:28:43.045553 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:43.045533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:43.176143 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:43.176109    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ggttg"]
Jul 27 17:28:43.180506 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:28:43.180472    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b376d75_6fb9_48ea_8621_e43b18a91e98.slice/crio-b4cf8ed677b1449b932301c577814733498eefce925edb606e9cfa26bb4c795f WatchSource:0}: Error finding container b4cf8ed677b1449b932301c577814733498eefce925edb606e9cfa26bb4c795f: Status 404 returned error can't find the container with id b4cf8ed677b1449b932301c577814733498eefce925edb606e9cfa26bb4c795f
Jul 27 17:28:43.190727 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:43.190697    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8wtkw"]
Jul 27 17:28:43.193219 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:28:43.193195    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f3ab66_e7e3_4565_8357_475969cf2579.slice/crio-b4f73ae3b2e0ed843aceecc49cd47c3a2df3e55aa3525a1cda0350add9652177 WatchSource:0}: Error finding container b4f73ae3b2e0ed843aceecc49cd47c3a2df3e55aa3525a1cda0350add9652177: Status 404 returned error can't find the container with id b4f73ae3b2e0ed843aceecc49cd47c3a2df3e55aa3525a1cda0350add9652177
Jul 27 17:28:43.416691 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:43.416602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8wtkw" event={"ID":"42f3ab66-e7e3-4565-8357-475969cf2579","Type":"ContainerStarted","Data":"b4f73ae3b2e0ed843aceecc49cd47c3a2df3e55aa3525a1cda0350add9652177"}
Jul 27 17:28:43.417614 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:43.417588    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ggttg" event={"ID":"3b376d75-6fb9-48ea-8621-e43b18a91e98","Type":"ContainerStarted","Data":"b4cf8ed677b1449b932301c577814733498eefce925edb606e9cfa26bb4c795f"}
Jul 27 17:28:47.429870 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:47.429786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ggttg" event={"ID":"3b376d75-6fb9-48ea-8621-e43b18a91e98","Type":"ContainerStarted","Data":"a9f37a8b0e7c93ef44aeba8a688f5b4477eb6d0dd2ae2d8f63ca5171acafa8fc"}
Jul 27 17:28:47.447289 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:47.447242    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ggttg" podStartSLOduration=66.507813325 podStartE2EDuration="1m10.447226593s" totalImagesPullingTime="3.939413268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:47.4468498 +0000 UTC m=+70.899470946" watchObservedRunningTime="2026-07-27 17:28:47.447226593 +0000 UTC m=+70.899847735"
Jul 27 17:28:49.436200 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:49.436152    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8wtkw" event={"ID":"42f3ab66-e7e3-4565-8357-475969cf2579","Type":"ContainerStarted","Data":"94a07e237baa565e41ed80e561ee007055f2ced05988426509f74f32ce4c9ebc"}
Jul 27 17:28:49.436619 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:49.436301    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:28:49.454100 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:28:49.454056    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8wtkw" podStartSLOduration=67.368514539 podStartE2EDuration="1m13.454042075s" totalImagesPullingTime="6.085527536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:49.453632883 +0000 UTC m=+72.906254029" watchObservedRunningTime="2026-07-27 17:28:49.454042075 +0000 UTC m=+72.906663199"
Jul 27 17:29:11.293889 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.293777    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"]
Jul 27 17:29:11.296133 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.296116    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.300891 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.300865    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 27 17:29:11.300891 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.300877    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 27 17:29:11.301197 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.301162    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 27 17:29:11.301312 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.301249    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mpdvs\""
Jul 27 17:29:11.302205 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.301989    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:29:11.309780 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.309764    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"]
Jul 27 17:29:11.390478 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.390444    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-2fmlt"]
Jul 27 17:29:11.392073 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.392054    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.394824 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.394802    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:29:11.394902 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.394847    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 17:29:11.395085 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.395070    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 17:29:11.395320 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.395302    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 17:29:11.395504 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.395490    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-69wbl\""
Jul 27 17:29:11.403528 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.403508    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 17:29:11.407997 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.407978    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-2fmlt"]
Jul 27 17:29:11.412435 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.412416    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989-serving-cert\") pod \"service-ca-operator-5d6c75f84f-zk57p\" (UID: \"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.412544 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.412453    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989-config\") pod \"service-ca-operator-5d6c75f84f-zk57p\" (UID: \"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.412544 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.412477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzrt\" (UniqueName: \"kubernetes.io/projected/9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989-kube-api-access-6hzrt\") pod \"service-ca-operator-5d6c75f84f-zk57p\" (UID: \"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.512784 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.512757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e83a9b52-aad4-4ffb-a228-896ef91a710c-trusted-ca\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.512911 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.512817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989-serving-cert\") pod \"service-ca-operator-5d6c75f84f-zk57p\" (UID: \"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.512911 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.512872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989-config\") pod \"service-ca-operator-5d6c75f84f-zk57p\" (UID: \"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.512911 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.512894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzrt\" (UniqueName: \"kubernetes.io/projected/9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989-kube-api-access-6hzrt\") pod \"service-ca-operator-5d6c75f84f-zk57p\" (UID: \"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.513063 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.512917    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e83a9b52-aad4-4ffb-a228-896ef91a710c-config\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.513063 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.512939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtx7l\" (UniqueName: \"kubernetes.io/projected/e83a9b52-aad4-4ffb-a228-896ef91a710c-kube-api-access-wtx7l\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.513063 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.513012    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e83a9b52-aad4-4ffb-a228-896ef91a710c-serving-cert\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.513454 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.513436    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989-config\") pod \"service-ca-operator-5d6c75f84f-zk57p\" (UID: \"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.514944 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.514927    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989-serving-cert\") pod \"service-ca-operator-5d6c75f84f-zk57p\" (UID: \"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.523829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.523807    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzrt\" (UniqueName: \"kubernetes.io/projected/9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989-kube-api-access-6hzrt\") pod \"service-ca-operator-5d6c75f84f-zk57p\" (UID: \"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.605243 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.605154    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"
Jul 27 17:29:11.614138 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.614120    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e83a9b52-aad4-4ffb-a228-896ef91a710c-config\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.614243 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.614150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtx7l\" (UniqueName: \"kubernetes.io/projected/e83a9b52-aad4-4ffb-a228-896ef91a710c-kube-api-access-wtx7l\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.614243 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.614199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e83a9b52-aad4-4ffb-a228-896ef91a710c-serving-cert\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.614243 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.614236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e83a9b52-aad4-4ffb-a228-896ef91a710c-trusted-ca\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.619547 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.616104    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e83a9b52-aad4-4ffb-a228-896ef91a710c-trusted-ca\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.619547 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.617258    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e83a9b52-aad4-4ffb-a228-896ef91a710c-config\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.621731 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.621707    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e83a9b52-aad4-4ffb-a228-896ef91a710c-serving-cert\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.624070 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.624052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtx7l\" (UniqueName: \"kubernetes.io/projected/e83a9b52-aad4-4ffb-a228-896ef91a710c-kube-api-access-wtx7l\") pod \"console-operator-584dcb6b54-2fmlt\" (UID: \"e83a9b52-aad4-4ffb-a228-896ef91a710c\") " pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.700528 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.700483    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:11.725652 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.725625    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p"]
Jul 27 17:29:11.728578 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:11.728546    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e27191f_4d7b_4bd5_a7b3_9e43cfbd4989.slice/crio-65b92e634bb27f48194d3338fbeb469c67503abb9a724c4dad139a5f057a084c WatchSource:0}: Error finding container 65b92e634bb27f48194d3338fbeb469c67503abb9a724c4dad139a5f057a084c: Status 404 returned error can't find the container with id 65b92e634bb27f48194d3338fbeb469c67503abb9a724c4dad139a5f057a084c
Jul 27 17:29:11.820890 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:11.820863    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-2fmlt"]
Jul 27 17:29:11.823485 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:11.823462    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83a9b52_aad4_4ffb_a228_896ef91a710c.slice/crio-feb2a1e8bd5df7ac9c97e600883f1c8f7e7eb71e711dc8c5a890e401822e1c5b WatchSource:0}: Error finding container feb2a1e8bd5df7ac9c97e600883f1c8f7e7eb71e711dc8c5a890e401822e1c5b: Status 404 returned error can't find the container with id feb2a1e8bd5df7ac9c97e600883f1c8f7e7eb71e711dc8c5a890e401822e1c5b
Jul 27 17:29:12.492199 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:12.492157    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt" event={"ID":"e83a9b52-aad4-4ffb-a228-896ef91a710c","Type":"ContainerStarted","Data":"feb2a1e8bd5df7ac9c97e600883f1c8f7e7eb71e711dc8c5a890e401822e1c5b"}
Jul 27 17:29:12.493387 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:12.493357    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p" event={"ID":"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989","Type":"ContainerStarted","Data":"65b92e634bb27f48194d3338fbeb469c67503abb9a724c4dad139a5f057a084c"}
Jul 27 17:29:14.500755 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:14.500730    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/0.log"
Jul 27 17:29:14.501162 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:14.500767    2564 generic.go:356] "Generic (PLEG): container finished" podID="e83a9b52-aad4-4ffb-a228-896ef91a710c" containerID="2069249ea01a1a051309d23e4bd13199d7bc404eac6d037a77c2b322866d632c" exitCode=255
Jul 27 17:29:14.501162 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:14.500804    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt" event={"ID":"e83a9b52-aad4-4ffb-a228-896ef91a710c","Type":"ContainerDied","Data":"2069249ea01a1a051309d23e4bd13199d7bc404eac6d037a77c2b322866d632c"}
Jul 27 17:29:14.501162 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:14.501057    2564 scope.go:117] "RemoveContainer" containerID="2069249ea01a1a051309d23e4bd13199d7bc404eac6d037a77c2b322866d632c"
Jul 27 17:29:14.502485 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:14.502458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p" event={"ID":"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989","Type":"ContainerStarted","Data":"61f964a58916aca1234fed32a1ddcf1ae51d58a728cd0a0e71e3a06038ffd5e1"}
Jul 27 17:29:14.536955 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:14.536896    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p" podStartSLOduration=0.96150306 podStartE2EDuration="3.536879311s" totalImagesPullingTime="2.575376251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:14.53594718 +0000 UTC m=+97.988568326" watchObservedRunningTime="2026-07-27 17:29:14.536879311 +0000 UTC m=+97.989500456"
Jul 27 17:29:14.840989 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:14.840908    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:29:14.840989 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:14.840945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:29:14.841150 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:14.841053    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:14.841150 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:14.841062    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:14.841150 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:14.841135    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls podName:332d404e-0e71-485a-8a0d-05951cf4752d nodeName:}" failed. No retries permitted until 2026-07-27 17:30:18.841116051 +0000 UTC m=+162.293737189 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls") pod "dns-default-xt257" (UID: "332d404e-0e71-485a-8a0d-05951cf4752d") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:14.841150 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:14.841065    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666b8c6bcb-f2nbx: secret "image-registry-tls" not found
Jul 27 17:29:14.841324 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:14.841214    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls podName:9abb877f-c36e-4524-bf95-241c606335ce nodeName:}" failed. No retries permitted until 2026-07-27 17:30:18.84120116 +0000 UTC m=+162.293822284 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls") pod "image-registry-666b8c6bcb-f2nbx" (UID: "9abb877f-c36e-4524-bf95-241c606335ce") : secret "image-registry-tls" not found
Jul 27 17:29:15.408461 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.408429    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ptt5w"]
Jul 27 17:29:15.410493 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.410478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.421021 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.420976    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 17:29:15.421196 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.421159    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 17:29:15.421273 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.421258    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 17:29:15.421665 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.421651    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 17:29:15.423023 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.422997    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cgk6t\""
Jul 27 17:29:15.440194 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.440165    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ptt5w"]
Jul 27 17:29:15.506973 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.506947    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:29:15.507378 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.507322    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/0.log"
Jul 27 17:29:15.507378 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.507353    2564 generic.go:356] "Generic (PLEG): container finished" podID="e83a9b52-aad4-4ffb-a228-896ef91a710c" containerID="cb0224eeb8efab2a8cde92a4d72058135d0a0a2aa81eb78abc0cf3160728f2e4" exitCode=255
Jul 27 17:29:15.507455 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.507384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt" event={"ID":"e83a9b52-aad4-4ffb-a228-896ef91a710c","Type":"ContainerDied","Data":"cb0224eeb8efab2a8cde92a4d72058135d0a0a2aa81eb78abc0cf3160728f2e4"}
Jul 27 17:29:15.507455 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.507450    2564 scope.go:117] "RemoveContainer" containerID="2069249ea01a1a051309d23e4bd13199d7bc404eac6d037a77c2b322866d632c"
Jul 27 17:29:15.507762 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.507744    2564 scope.go:117] "RemoveContainer" containerID="cb0224eeb8efab2a8cde92a4d72058135d0a0a2aa81eb78abc0cf3160728f2e4"
Jul 27 17:29:15.507929 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:15.507906    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-2fmlt_openshift-console-operator(e83a9b52-aad4-4ffb-a228-896ef91a710c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt" podUID="e83a9b52-aad4-4ffb-a228-896ef91a710c"
Jul 27 17:29:15.547241 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.547205    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/245f72c6-ba0b-47c0-9631-6966ca016a69-data-volume\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.547382 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.547256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hr86\" (UniqueName: \"kubernetes.io/projected/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-api-access-2hr86\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.547382 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.547306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/245f72c6-ba0b-47c0-9631-6966ca016a69-crio-socket\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.547382 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.547328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.547382 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.547356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.648319 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.648281    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/245f72c6-ba0b-47c0-9631-6966ca016a69-data-volume\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.648494 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.648335    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hr86\" (UniqueName: \"kubernetes.io/projected/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-api-access-2hr86\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.648494 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.648481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/245f72c6-ba0b-47c0-9631-6966ca016a69-crio-socket\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.648628 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.648526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.648628 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.648586    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/245f72c6-ba0b-47c0-9631-6966ca016a69-crio-socket\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.648628 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.648601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.648762 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:15.648668    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:15.648762 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.648675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/245f72c6-ba0b-47c0-9631-6966ca016a69-data-volume\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:15.648762 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:15.648720    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:15.648762 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:15.648760    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:16.148738812 +0000 UTC m=+99.601360001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:15.648907 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:15.648783    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:16.14877226 +0000 UTC m=+99.601393384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:15.659461 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:15.659404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hr86\" (UniqueName: \"kubernetes.io/projected/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-api-access-2hr86\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:16.152854 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:16.152809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:16.152854 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:16.152854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:16.153069 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:16.152932    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:16.153069 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:16.152962    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:16.153069 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:16.153014    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:17.152999094 +0000 UTC m=+100.605620231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:16.153069 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:16.153028    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:17.153022049 +0000 UTC m=+100.605643173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:16.512516 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:16.512483    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:29:16.512906 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:16.512793    2564 scope.go:117] "RemoveContainer" containerID="cb0224eeb8efab2a8cde92a4d72058135d0a0a2aa81eb78abc0cf3160728f2e4"
Jul 27 17:29:16.512950 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:16.512934    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-2fmlt_openshift-console-operator(e83a9b52-aad4-4ffb-a228-896ef91a710c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt" podUID="e83a9b52-aad4-4ffb-a228-896ef91a710c"
Jul 27 17:29:17.160667 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.160635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:17.160667 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.160667    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:17.160887 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:17.160747    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:17.160887 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:17.160782    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:17.160887 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:17.160808    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:19.160793116 +0000 UTC m=+102.613414240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:17.160887 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:17.160823    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:19.16081676 +0000 UTC m=+102.613437884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:17.195741 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.195720    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2lkcp_df9a0647-8600-491a-8042-186017147267/dns-node-resolver/0.log"
Jul 27 17:29:17.947020 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.946985    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-7twnx"]
Jul 27 17:29:17.949502 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.949486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:17.953131 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.953105    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 27 17:29:17.953331 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.953111    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6tznz\""
Jul 27 17:29:17.954163 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.954144    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 27 17:29:17.954488 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.954470    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 27 17:29:17.954702 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.954687    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 27 17:29:17.958598 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:17.958566    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-7twnx"]
Jul 27 17:29:18.067286 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.067253    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5ba4e63c-620f-4514-adc1-77909816b4a9-signing-cabundle\") pod \"service-ca-658d7b7c78-7twnx\" (UID: \"5ba4e63c-620f-4514-adc1-77909816b4a9\") " pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.067435 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.067314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5ba4e63c-620f-4514-adc1-77909816b4a9-signing-key\") pod \"service-ca-658d7b7c78-7twnx\" (UID: \"5ba4e63c-620f-4514-adc1-77909816b4a9\") " pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.067491 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.067467    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcnxh\" (UniqueName: \"kubernetes.io/projected/5ba4e63c-620f-4514-adc1-77909816b4a9-kube-api-access-mcnxh\") pod \"service-ca-658d7b7c78-7twnx\" (UID: \"5ba4e63c-620f-4514-adc1-77909816b4a9\") " pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.168561 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.168528    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcnxh\" (UniqueName: \"kubernetes.io/projected/5ba4e63c-620f-4514-adc1-77909816b4a9-kube-api-access-mcnxh\") pod \"service-ca-658d7b7c78-7twnx\" (UID: \"5ba4e63c-620f-4514-adc1-77909816b4a9\") " pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.168698 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.168569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5ba4e63c-620f-4514-adc1-77909816b4a9-signing-cabundle\") pod \"service-ca-658d7b7c78-7twnx\" (UID: \"5ba4e63c-620f-4514-adc1-77909816b4a9\") " pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.168698 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.168621    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5ba4e63c-620f-4514-adc1-77909816b4a9-signing-key\") pod \"service-ca-658d7b7c78-7twnx\" (UID: \"5ba4e63c-620f-4514-adc1-77909816b4a9\") " pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.169277 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.169253    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5ba4e63c-620f-4514-adc1-77909816b4a9-signing-cabundle\") pod \"service-ca-658d7b7c78-7twnx\" (UID: \"5ba4e63c-620f-4514-adc1-77909816b4a9\") " pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.170917 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.170890    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5ba4e63c-620f-4514-adc1-77909816b4a9-signing-key\") pod \"service-ca-658d7b7c78-7twnx\" (UID: \"5ba4e63c-620f-4514-adc1-77909816b4a9\") " pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.177433 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.177411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcnxh\" (UniqueName: \"kubernetes.io/projected/5ba4e63c-620f-4514-adc1-77909816b4a9-kube-api-access-mcnxh\") pod \"service-ca-658d7b7c78-7twnx\" (UID: \"5ba4e63c-620f-4514-adc1-77909816b4a9\") " pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.261397 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.261370    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-7twnx"
Jul 27 17:29:18.384104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.384068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-7twnx"]
Jul 27 17:29:18.387367 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:18.387341    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba4e63c_620f_4514_adc1_77909816b4a9.slice/crio-9f2b1063de517d6ae089e5b8d9951fa999846752db86bf6195cdf651f8a8c87b WatchSource:0}: Error finding container 9f2b1063de517d6ae089e5b8d9951fa999846752db86bf6195cdf651f8a8c87b: Status 404 returned error can't find the container with id 9f2b1063de517d6ae089e5b8d9951fa999846752db86bf6195cdf651f8a8c87b
Jul 27 17:29:18.393489 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.393472    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v8rh6_9c64d459-de22-4e66-a489-4c7ac0a62897/node-ca/0.log"
Jul 27 17:29:18.520434 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.520349    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-7twnx" event={"ID":"5ba4e63c-620f-4514-adc1-77909816b4a9","Type":"ContainerStarted","Data":"f16ddff719681a3cc1acb7fff4c6507d9a9b93e7f91da25bff0412d2c840e9da"}
Jul 27 17:29:18.520434 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.520416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-7twnx" event={"ID":"5ba4e63c-620f-4514-adc1-77909816b4a9","Type":"ContainerStarted","Data":"9f2b1063de517d6ae089e5b8d9951fa999846752db86bf6195cdf651f8a8c87b"}
Jul 27 17:29:18.538760 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:18.538719    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-7twnx" podStartSLOduration=1.538705199 podStartE2EDuration="1.538705199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:18.538214457 +0000 UTC m=+101.990835600" watchObservedRunningTime="2026-07-27 17:29:18.538705199 +0000 UTC m=+101.991326346"
Jul 27 17:29:19.176988 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:19.176950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:19.177402 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:19.176998    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:19.177402 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:19.177090    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:19.177402 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:19.177137    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:19.177402 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:19.177193    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:23.177153682 +0000 UTC m=+106.629774820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:19.177402 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:19.177219    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:23.177206437 +0000 UTC m=+106.629827591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:19.393712 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:19.393679    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mjj85_7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac/serve-healthcheck-canary/0.log"
Jul 27 17:29:20.440592 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:20.440565    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8wtkw"
Jul 27 17:29:21.700732 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:21.700698    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:21.701351 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:21.700773    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:21.701351 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:21.701245    2564 scope.go:117] "RemoveContainer" containerID="cb0224eeb8efab2a8cde92a4d72058135d0a0a2aa81eb78abc0cf3160728f2e4"
Jul 27 17:29:21.701477 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:21.701443    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-2fmlt_openshift-console-operator(e83a9b52-aad4-4ffb-a228-896ef91a710c)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt" podUID="e83a9b52-aad4-4ffb-a228-896ef91a710c"
Jul 27 17:29:23.209897 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:23.209857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:23.209897 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:23.209901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:23.210464 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:23.209973    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:23.210464 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:23.210037    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:31.210021025 +0000 UTC m=+114.662642149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:23.210464 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:23.210415    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:23.210630 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:23.210517    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls podName:245f72c6-ba0b-47c0-9631-6966ca016a69 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:31.210486364 +0000 UTC m=+114.663107504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ptt5w" (UID: "245f72c6-ba0b-47c0-9631-6966ca016a69") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:31.274774 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:31.274732    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:31.274774 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:31.274777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:31.275317 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:31.275296    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/245f72c6-ba0b-47c0-9631-6966ca016a69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:31.277103 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:31.277081    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/245f72c6-ba0b-47c0-9631-6966ca016a69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ptt5w\" (UID: \"245f72c6-ba0b-47c0-9631-6966ca016a69\") " pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:31.318614 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:31.318585    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ptt5w"
Jul 27 17:29:31.444324 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:31.444291    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ptt5w"]
Jul 27 17:29:31.447289 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:31.447264    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245f72c6_ba0b_47c0_9631_6966ca016a69.slice/crio-d9752b15010844af4dc77d6dd40db1c6123ce4d3554c7c7af386b33f872fa349 WatchSource:0}: Error finding container d9752b15010844af4dc77d6dd40db1c6123ce4d3554c7c7af386b33f872fa349: Status 404 returned error can't find the container with id d9752b15010844af4dc77d6dd40db1c6123ce4d3554c7c7af386b33f872fa349
Jul 27 17:29:31.560455 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:31.560377    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ptt5w" event={"ID":"245f72c6-ba0b-47c0-9631-6966ca016a69","Type":"ContainerStarted","Data":"d9752b15010844af4dc77d6dd40db1c6123ce4d3554c7c7af386b33f872fa349"}
Jul 27 17:29:34.114774 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:34.114740    2564 scope.go:117] "RemoveContainer" containerID="cb0224eeb8efab2a8cde92a4d72058135d0a0a2aa81eb78abc0cf3160728f2e4"
Jul 27 17:29:34.572196 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:34.572149    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:29:34.572362 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:34.572265    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt" event={"ID":"e83a9b52-aad4-4ffb-a228-896ef91a710c","Type":"ContainerStarted","Data":"2d95713dae373077d1dfb583fb8c954e1776bf71e5e5e6c205d042d3c6610952"}
Jul 27 17:29:34.572671 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:34.572640    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:34.598451 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:34.598393    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt" podStartSLOduration=21.117863101 podStartE2EDuration="23.598373172s" totalImagesPullingTime="2.480510071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:34.597017087 +0000 UTC m=+118.049638237" watchObservedRunningTime="2026-07-27 17:29:34.598373172 +0000 UTC m=+118.050994320"
Jul 27 17:29:34.973491 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:34.973461    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-2fmlt"
Jul 27 17:29:38.586453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:38.586416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ptt5w" event={"ID":"245f72c6-ba0b-47c0-9631-6966ca016a69","Type":"ContainerStarted","Data":"4f3b7b937c3a0f688fb7ec843e07186b8062432bae2dca6bbee1c7a396ef0f0d"}
Jul 27 17:29:39.298129 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.298090    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"]
Jul 27 17:29:39.299892 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.299872    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"
Jul 27 17:29:39.305407 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.305373    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 17:29:39.305697 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.305682    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 17:29:39.305795 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.305773    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-l7lq9\""
Jul 27 17:29:39.306839 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.306818    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 17:29:39.306970 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.306953    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 17:29:39.324794 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.324771    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"]
Jul 27 17:29:39.334102 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.334048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bpl\" (UniqueName: \"kubernetes.io/projected/23ba2a71-826d-4e96-babf-7000cf857aac-kube-api-access-72bpl\") pod \"managed-serviceaccount-addon-agent-95d4c9f76-c88w9\" (UID: \"23ba2a71-826d-4e96-babf-7000cf857aac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"
Jul 27 17:29:39.334256 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.334201    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ba2a71-826d-4e96-babf-7000cf857aac-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-95d4c9f76-c88w9\" (UID: \"23ba2a71-826d-4e96-babf-7000cf857aac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"
Jul 27 17:29:39.388846 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.388815    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-865bb"]
Jul 27 17:29:39.390523 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.390507    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb"
Jul 27 17:29:39.392071 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.392051    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-nlscd"]
Jul 27 17:29:39.393823 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.393808    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-nlscd"
Jul 27 17:29:39.394581 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.394561    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 27 17:29:39.394812 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.394795    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6swfz\""
Jul 27 17:29:39.394899 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.394834    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 27 17:29:39.396581 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.396563    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 17:29:39.396842 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.396823    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kkrln\""
Jul 27 17:29:39.397313 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.397146    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 17:29:39.408768 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.408746    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-865bb"]
Jul 27 17:29:39.411218 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.411172    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-nlscd"]
Jul 27 17:29:39.434969 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.434942    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e73495df-4bcc-4b05-883e-a539562031b4-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-865bb\" (UID: \"e73495df-4bcc-4b05-883e-a539562031b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb"
Jul 27 17:29:39.435123 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.434982    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bpl\" (UniqueName: \"kubernetes.io/projected/23ba2a71-826d-4e96-babf-7000cf857aac-kube-api-access-72bpl\") pod \"managed-serviceaccount-addon-agent-95d4c9f76-c88w9\" (UID: \"23ba2a71-826d-4e96-babf-7000cf857aac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"
Jul 27 17:29:39.435123 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.435009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drw9w\" (UniqueName: \"kubernetes.io/projected/ee681144-a38e-4964-9c30-604dfd4e6d71-kube-api-access-drw9w\") pod \"downloads-7c97dc4555-nlscd\" (UID: \"ee681144-a38e-4964-9c30-604dfd4e6d71\") " pod="openshift-console/downloads-7c97dc4555-nlscd"
Jul 27 17:29:39.435257 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.435171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e73495df-4bcc-4b05-883e-a539562031b4-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-865bb\" (UID: \"e73495df-4bcc-4b05-883e-a539562031b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb"
Jul 27 17:29:39.435307 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.435261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ba2a71-826d-4e96-babf-7000cf857aac-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-95d4c9f76-c88w9\" (UID: \"23ba2a71-826d-4e96-babf-7000cf857aac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"
Jul 27 17:29:39.437577 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.437554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ba2a71-826d-4e96-babf-7000cf857aac-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-95d4c9f76-c88w9\" (UID: \"23ba2a71-826d-4e96-babf-7000cf857aac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"
Jul 27 17:29:39.453646 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.453619    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bpl\" (UniqueName: \"kubernetes.io/projected/23ba2a71-826d-4e96-babf-7000cf857aac-kube-api-access-72bpl\") pod \"managed-serviceaccount-addon-agent-95d4c9f76-c88w9\" (UID: \"23ba2a71-826d-4e96-babf-7000cf857aac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"
Jul 27 17:29:39.535956 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.535927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e73495df-4bcc-4b05-883e-a539562031b4-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-865bb\" (UID: \"e73495df-4bcc-4b05-883e-a539562031b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb"
Jul 27 17:29:39.536138 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.535974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drw9w\" (UniqueName: \"kubernetes.io/projected/ee681144-a38e-4964-9c30-604dfd4e6d71-kube-api-access-drw9w\") pod \"downloads-7c97dc4555-nlscd\" (UID: \"ee681144-a38e-4964-9c30-604dfd4e6d71\") " pod="openshift-console/downloads-7c97dc4555-nlscd"
Jul 27 17:29:39.536138 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.536017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e73495df-4bcc-4b05-883e-a539562031b4-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-865bb\" (UID: \"e73495df-4bcc-4b05-883e-a539562031b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb"
Jul 27 17:29:39.536601 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.536574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e73495df-4bcc-4b05-883e-a539562031b4-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-865bb\" (UID: \"e73495df-4bcc-4b05-883e-a539562031b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb"
Jul 27 17:29:39.538385 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.538365    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e73495df-4bcc-4b05-883e-a539562031b4-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-865bb\" (UID: \"e73495df-4bcc-4b05-883e-a539562031b4\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb"
Jul 27 17:29:39.546640 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.546617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drw9w\" (UniqueName: \"kubernetes.io/projected/ee681144-a38e-4964-9c30-604dfd4e6d71-kube-api-access-drw9w\") pod \"downloads-7c97dc4555-nlscd\" (UID: \"ee681144-a38e-4964-9c30-604dfd4e6d71\") " pod="openshift-console/downloads-7c97dc4555-nlscd"
Jul 27 17:29:39.616077 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.616006    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"
Jul 27 17:29:39.700349 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.700314    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb"
Jul 27 17:29:39.705499 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.705471    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-nlscd"
Jul 27 17:29:39.737525 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.737489    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9"]
Jul 27 17:29:39.740911 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:39.740870    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23ba2a71_826d_4e96_babf_7000cf857aac.slice/crio-b43320092155514c50bd21544588c370a3b62712657448fac406fb339207cd66 WatchSource:0}: Error finding container b43320092155514c50bd21544588c370a3b62712657448fac406fb339207cd66: Status 404 returned error can't find the container with id b43320092155514c50bd21544588c370a3b62712657448fac406fb339207cd66
Jul 27 17:29:39.838481 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.838447    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-865bb"]
Jul 27 17:29:39.841372 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:39.841342    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73495df_4bcc_4b05_883e_a539562031b4.slice/crio-18e942203c41d31b597f18e49a80e0fb37a5a32bf3300aa98fa189fbfe086844 WatchSource:0}: Error finding container 18e942203c41d31b597f18e49a80e0fb37a5a32bf3300aa98fa189fbfe086844: Status 404 returned error can't find the container with id 18e942203c41d31b597f18e49a80e0fb37a5a32bf3300aa98fa189fbfe086844
Jul 27 17:29:39.855757 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:39.855735    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-nlscd"]
Jul 27 17:29:39.858192 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:39.858156    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee681144_a38e_4964_9c30_604dfd4e6d71.slice/crio-cf43dc0df2a045ee1c66914c99c8b07eabe01735458cebca6c1fa47f929d1e67 WatchSource:0}: Error finding container cf43dc0df2a045ee1c66914c99c8b07eabe01735458cebca6c1fa47f929d1e67: Status 404 returned error can't find the container with id cf43dc0df2a045ee1c66914c99c8b07eabe01735458cebca6c1fa47f929d1e67
Jul 27 17:29:40.593795 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:40.593760    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9" event={"ID":"23ba2a71-826d-4e96-babf-7000cf857aac","Type":"ContainerStarted","Data":"b43320092155514c50bd21544588c370a3b62712657448fac406fb339207cd66"}
Jul 27 17:29:40.595052 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:40.595019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-nlscd" event={"ID":"ee681144-a38e-4964-9c30-604dfd4e6d71","Type":"ContainerStarted","Data":"cf43dc0df2a045ee1c66914c99c8b07eabe01735458cebca6c1fa47f929d1e67"}
Jul 27 17:29:40.596305 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:40.596275    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb" event={"ID":"e73495df-4bcc-4b05-883e-a539562031b4","Type":"ContainerStarted","Data":"18e942203c41d31b597f18e49a80e0fb37a5a32bf3300aa98fa189fbfe086844"}
Jul 27 17:29:43.616369 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:43.616330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ptt5w" event={"ID":"245f72c6-ba0b-47c0-9631-6966ca016a69","Type":"ContainerStarted","Data":"1072f8d0410f2963060f1d01b24b235b5f18e4d83572b94b903a53c2dd76ee4c"}
Jul 27 17:29:44.075635 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.075592    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f595bfff8-lzmnt"]
Jul 27 17:29:44.082386 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.082359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.087241 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.087006    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 17:29:44.087241 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.087060    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 17:29:44.087908 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.087543    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 17:29:44.087908 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.087652    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 17:29:44.087908 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.087743    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 17:29:44.087908 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.087873    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d8nfx\""
Jul 27 17:29:44.092584 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.092236    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f595bfff8-lzmnt"]
Jul 27 17:29:44.177001 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.176899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-service-ca\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.177001 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.176952    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-config\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.177260 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.177015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-oauth-serving-cert\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.177260 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.177041    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24mr\" (UniqueName: \"kubernetes.io/projected/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-kube-api-access-c24mr\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.177260 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.177068    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-serving-cert\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.177260 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.177092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-oauth-config\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.278758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.277729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-oauth-serving-cert\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.278758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.277780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24mr\" (UniqueName: \"kubernetes.io/projected/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-kube-api-access-c24mr\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.278758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.277819    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-serving-cert\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.278758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.277844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-oauth-config\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.278758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.277909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-service-ca\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.278758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.277941    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-config\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.278758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.278710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-config\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.279274 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.279250    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-service-ca\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.279767 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.279733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-oauth-serving-cert\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.281074 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.281041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-oauth-config\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.283542 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.283514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-serving-cert\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.294671 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.294624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24mr\" (UniqueName: \"kubernetes.io/projected/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-kube-api-access-c24mr\") pod \"console-6f595bfff8-lzmnt\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") " pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:44.396168 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:44.396051    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:45.503826 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.503761    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f595bfff8-lzmnt"]
Jul 27 17:29:45.626813 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.626707    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb" event={"ID":"e73495df-4bcc-4b05-883e-a539562031b4","Type":"ContainerStarted","Data":"652ef819e7c93c5a8bfb09dbe6f1a0c16ac475466ed3f29d6eafcb1257d3aa9d"}
Jul 27 17:29:45.628630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.628571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9" event={"ID":"23ba2a71-826d-4e96-babf-7000cf857aac","Type":"ContainerStarted","Data":"57e264e29bd0ca55f935a2a7ad7d6cb1262aeaa3ee986e5010cb3c31a6bb5359"}
Jul 27 17:29:45.629959 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.629912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f595bfff8-lzmnt" event={"ID":"7ecc8a0e-47c6-458f-8862-c787e6d59f1e","Type":"ContainerStarted","Data":"a341b23916179438c0f3289baca9836adb1b61fd7226996add566e2016b1f8ac"}
Jul 27 17:29:45.645126 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.645064    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-865bb" podStartSLOduration=1.135794628 podStartE2EDuration="6.645045485s" totalImagesPullingTime="5.509250857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:45.644201364 +0000 UTC m=+129.096822505" watchObservedRunningTime="2026-07-27 17:29:45.645045485 +0000 UTC m=+129.097666628"
Jul 27 17:29:45.662340 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.662282    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95d4c9f76-c88w9" podStartSLOduration=0.978979991 podStartE2EDuration="6.662260318s" totalImagesPullingTime="5.683280327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:45.660981665 +0000 UTC m=+129.113602811" watchObservedRunningTime="2026-07-27 17:29:45.662260318 +0000 UTC m=+129.114881465"
Jul 27 17:29:45.888741 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.888607    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b6d68669d-jrdfl"]
Jul 27 17:29:45.893686 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.893659    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:45.904724 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.904647    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6d68669d-jrdfl"]
Jul 27 17:29:45.904865 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.904802    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 17:29:45.994061 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.994010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbtp\" (UniqueName: \"kubernetes.io/projected/c8b4dec9-4516-48fd-ad49-5f4456c8288d-kube-api-access-fjbtp\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:45.994061 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.994063    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-serving-cert\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:45.994347 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.994089    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-service-ca\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:45.994347 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.994118    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-oauth-config\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:45.994347 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.994171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-oauth-serving-cert\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:45.994347 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.994292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-trusted-ca-bundle\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:45.994347 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:45.994342    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-config\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.095536 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.095495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-config\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.095712 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.095597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbtp\" (UniqueName: \"kubernetes.io/projected/c8b4dec9-4516-48fd-ad49-5f4456c8288d-kube-api-access-fjbtp\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.095712 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.095629    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-serving-cert\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.095712 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.095656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-service-ca\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.095712 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.095685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-oauth-config\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.095884 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.095721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-oauth-serving-cert\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.095884 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.095783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-trusted-ca-bundle\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.097866 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.096738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-service-ca\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.097866 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.097000    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-trusted-ca-bundle\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.097866 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.097023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-oauth-serving-cert\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.097866 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.097214    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-config\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.099418 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.099379    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-serving-cert\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.101175 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.099392    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-oauth-config\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.106747 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.106721    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbtp\" (UniqueName: \"kubernetes.io/projected/c8b4dec9-4516-48fd-ad49-5f4456c8288d-kube-api-access-fjbtp\") pod \"console-5b6d68669d-jrdfl\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") " pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.205500 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.205422    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:46.802042 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.801979    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:29:46.804884 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.804854    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b84e87f-4fbf-471c-bf2f-2c364e4d9a82-metrics-certs\") pod \"network-metrics-daemon-flnfk\" (UID: \"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82\") " pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:29:46.945912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.945861    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p44n4\""
Jul 27 17:29:46.950624 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:46.950593    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flnfk"
Jul 27 17:29:47.842799 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:47.842755    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6d68669d-jrdfl"]
Jul 27 17:29:47.847839 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:47.847801    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4 WatchSource:0}: Error finding container 15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4: Status 404 returned error can't find the container with id 15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4
Jul 27 17:29:47.855610 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:47.855588    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flnfk"]
Jul 27 17:29:47.860213 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:47.859169    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b84e87f_4fbf_471c_bf2f_2c364e4d9a82.slice/crio-740de293793b1d2d27936b136ec8e121f4488f7411b3c7b6712c85f5014c1e2a WatchSource:0}: Error finding container 740de293793b1d2d27936b136ec8e121f4488f7411b3c7b6712c85f5014c1e2a: Status 404 returned error can't find the container with id 740de293793b1d2d27936b136ec8e121f4488f7411b3c7b6712c85f5014c1e2a
Jul 27 17:29:48.644218 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:48.643631    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ptt5w" event={"ID":"245f72c6-ba0b-47c0-9631-6966ca016a69","Type":"ContainerStarted","Data":"a99bed297e31e7077a7436cd5372c91eaa4b5f0bf443370d1ab9c8807c648d3c"}
Jul 27 17:29:48.646431 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:48.646357    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6d68669d-jrdfl" event={"ID":"c8b4dec9-4516-48fd-ad49-5f4456c8288d","Type":"ContainerStarted","Data":"15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4"}
Jul 27 17:29:48.647843 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:48.647723    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flnfk" event={"ID":"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82","Type":"ContainerStarted","Data":"740de293793b1d2d27936b136ec8e121f4488f7411b3c7b6712c85f5014c1e2a"}
Jul 27 17:29:48.667412 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:48.667279    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ptt5w" podStartSLOduration=17.495344356 podStartE2EDuration="33.667262152s" totalImagesPullingTime="16.171917796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:29:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:48.664896066 +0000 UTC m=+132.117517215" watchObservedRunningTime="2026-07-27 17:29:48.667262152 +0000 UTC m=+132.119883303"
Jul 27 17:29:51.662652 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:51.662559    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flnfk" event={"ID":"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82","Type":"ContainerStarted","Data":"30d73d594752aa1fb9a60ddb5ec1f26509e482031785ed0d5fb90973b6435aab"}
Jul 27 17:29:51.662652 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:51.662632    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flnfk" event={"ID":"6b84e87f-4fbf-471c-bf2f-2c364e4d9a82","Type":"ContainerStarted","Data":"5067c36c21167066194cbb613260e44be3f33950d8c4183287019faca93dc132"}
Jul 27 17:29:51.664386 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:51.664353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6d68669d-jrdfl" event={"ID":"c8b4dec9-4516-48fd-ad49-5f4456c8288d","Type":"ContainerStarted","Data":"a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09"}
Jul 27 17:29:51.666226 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:51.666177    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f595bfff8-lzmnt" event={"ID":"7ecc8a0e-47c6-458f-8862-c787e6d59f1e","Type":"ContainerStarted","Data":"63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397"}
Jul 27 17:29:51.684408 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:51.684355    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-flnfk" podStartSLOduration=131.221087125 podStartE2EDuration="2m14.684335832s" totalImagesPullingTime="3.463248707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:51.682865412 +0000 UTC m=+135.135486559" watchObservedRunningTime="2026-07-27 17:29:51.684335832 +0000 UTC m=+135.136956975"
Jul 27 17:29:51.702873 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:51.702821    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b6d68669d-jrdfl" podStartSLOduration=3.816157203 podStartE2EDuration="6.702802337s" totalImagesPullingTime="2.886645134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:51.702001675 +0000 UTC m=+135.154622824" watchObservedRunningTime="2026-07-27 17:29:51.702802337 +0000 UTC m=+135.155423474"
Jul 27 17:29:51.721991 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:51.721939    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f595bfff8-lzmnt" podStartSLOduration=2.501278185 podStartE2EDuration="7.721921579s" totalImagesPullingTime="5.220643394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:51.721576915 +0000 UTC m=+135.174198063" watchObservedRunningTime="2026-07-27 17:29:51.721921579 +0000 UTC m=+135.174542727"
Jul 27 17:29:53.551627 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.550867    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-629cq"]
Jul 27 17:29:53.555811 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.555790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.558673 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.558645    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 17:29:53.559089 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.559067    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lknvb\""
Jul 27 17:29:53.560055 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.559374    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 17:29:53.561167 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.561150    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 17:29:53.561914 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.561368    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 17:29:53.561914 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.561582    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 17:29:53.561914 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.561779    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 17:29:53.576863 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.576840    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"]
Jul 27 17:29:53.580450 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.580429    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.584050 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.584019    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m72ng\""
Jul 27 17:29:53.584275 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.584258    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 17:29:53.584373 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.584316    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 17:29:53.584860 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.584697    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 17:29:53.600031 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.599969    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"]
Jul 27 17:29:53.670296 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670265    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-wtmp\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.670296 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-textfile\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.670548 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4587b5df-0686-4ef4-b0e7-829add7246f4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.670548 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670373    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.670548 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-accelerators-collector-config\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.670548 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rjx\" (UniqueName: \"kubernetes.io/projected/1a09cd19-3146-4617-961e-614b1edf601b-kube-api-access-x8rjx\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.670548 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670457    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.670548 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670488    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4587b5df-0686-4ef4-b0e7-829add7246f4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.670548 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpmqj\" (UniqueName: \"kubernetes.io/projected/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-api-access-mpmqj\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.670891 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a09cd19-3146-4617-961e-614b1edf601b-root\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.670891 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670613    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.670891 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670647    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-tls\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.670891 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a09cd19-3146-4617-961e-614b1edf601b-sys\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.670891 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670736    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a09cd19-3146-4617-961e-614b1edf601b-metrics-client-ca\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.670891 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.670758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.771404 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771369    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4587b5df-0686-4ef4-b0e7-829add7246f4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.771596 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpmqj\" (UniqueName: \"kubernetes.io/projected/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-api-access-mpmqj\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.771596 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771476    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a09cd19-3146-4617-961e-614b1edf601b-root\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.771596 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.771596 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-tls\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.771596 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a09cd19-3146-4617-961e-614b1edf601b-sys\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.771912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771611    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a09cd19-3146-4617-961e-614b1edf601b-metrics-client-ca\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.771912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.771912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-wtmp\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.771912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-textfile\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.771912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4587b5df-0686-4ef4-b0e7-829add7246f4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.771912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.771912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-accelerators-collector-config\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.771912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rjx\" (UniqueName: \"kubernetes.io/projected/1a09cd19-3146-4617-961e-614b1edf601b-kube-api-access-x8rjx\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.771912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.771904    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.772390 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.772055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-wtmp\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.772390 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.772270    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4587b5df-0686-4ef4-b0e7-829add7246f4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.772499 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.772422    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-textfile\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.772499 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.772485    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a09cd19-3146-4617-961e-614b1edf601b-sys\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.772605 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:53.772568    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 17:29:53.772656 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:29:53.772625    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-tls podName:1a09cd19-3146-4617-961e-614b1edf601b nodeName:}" failed. No retries permitted until 2026-07-27 17:29:54.272607262 +0000 UTC m=+137.725228399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-tls") pod "node-exporter-629cq" (UID: "1a09cd19-3146-4617-961e-614b1edf601b") : secret "node-exporter-tls" not found
Jul 27 17:29:53.775956 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.773308    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.775956 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.773704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a09cd19-3146-4617-961e-614b1edf601b-root\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.775956 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.774028    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-accelerators-collector-config\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.775956 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.774781    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4587b5df-0686-4ef4-b0e7-829add7246f4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.775956 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.775454    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a09cd19-3146-4617-961e-614b1edf601b-metrics-client-ca\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.775956 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.775913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.779967 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.776693    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.780202 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.780160    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.783537 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.783490    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rjx\" (UniqueName: \"kubernetes.io/projected/1a09cd19-3146-4617-961e-614b1edf601b-kube-api-access-x8rjx\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:53.783971 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.783938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpmqj\" (UniqueName: \"kubernetes.io/projected/4587b5df-0686-4ef4-b0e7-829add7246f4-kube-api-access-mpmqj\") pod \"kube-state-metrics-56d9f944b4-45xpz\" (UID: \"4587b5df-0686-4ef4-b0e7-829add7246f4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:53.894122 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:53.893812    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"
Jul 27 17:29:54.276925 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:54.276890    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-tls\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:54.279555 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:54.279523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a09cd19-3146-4617-961e-614b1edf601b-node-exporter-tls\") pod \"node-exporter-629cq\" (UID: \"1a09cd19-3146-4617-961e-614b1edf601b\") " pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:54.396867 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:54.396829    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:54.397051 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:54.396934    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:29:54.398355 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:54.398327    2564 patch_prober.go:28] interesting pod/console-6f595bfff8-lzmnt container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body=
Jul 27 17:29:54.398491 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:54.398378    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6f595bfff8-lzmnt" podUID="7ecc8a0e-47c6-458f-8862-c787e6d59f1e" containerName="console" probeResult="failure" output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused"
Jul 27 17:29:54.467832 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:54.467777    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-629cq"
Jul 27 17:29:56.206153 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:56.206113    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:56.206620 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:56.206206    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:29:56.207624 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:56.207593    2564 patch_prober.go:28] interesting pod/console-5b6d68669d-jrdfl container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.17:8443/health\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body=
Jul 27 17:29:56.207750 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:56.207648    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5b6d68669d-jrdfl" podUID="c8b4dec9-4516-48fd-ad49-5f4456c8288d" containerName="console" probeResult="failure" output="Get \"https://10.132.0.17:8443/health\": dial tcp 10.132.0.17:8443: connect: connection refused"
Jul 27 17:29:57.920214 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:57.920164    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bd6857454-wqt6s"]
Jul 27 17:29:57.930606 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:57.930582    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:57.934949 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:57.934879    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 27 17:29:57.934949 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:57.934936    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gqklj\""
Jul 27 17:29:57.935127 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:57.935059    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ce4oldpmetsq0\""
Jul 27 17:29:57.935207 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:57.935127    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 17:29:57.935312 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:57.935287    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 27 17:29:57.935376 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:57.935360    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 27 17:29:57.935698 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:57.935673    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bd6857454-wqt6s"]
Jul 27 17:29:58.010874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.010837    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd54k\" (UniqueName: \"kubernetes.io/projected/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-kube-api-access-fd54k\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.011062 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.010977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.011062 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.011018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-audit-log\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.011062 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.011041    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-client-ca-bundle\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.011224 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.011166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-secret-metrics-server-client-certs\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.011274 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.011243    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-metrics-server-audit-profiles\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.011320 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.011278    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-secret-metrics-server-tls\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.111912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.111871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.112105 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.111924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-audit-log\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.112105 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.111954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-client-ca-bundle\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.112105 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.112017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-secret-metrics-server-client-certs\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.112105 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.112053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-metrics-server-audit-profiles\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.112105 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.112084    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-secret-metrics-server-tls\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.112485 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.112113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd54k\" (UniqueName: \"kubernetes.io/projected/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-kube-api-access-fd54k\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.115039 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.112369    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-audit-log\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.115039 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.113791    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.115039 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.114514    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-metrics-server-audit-profiles\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.119360 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.115702    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-client-ca-bundle\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.119360 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.116427    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-secret-metrics-server-client-certs\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.119360 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.117013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-secret-metrics-server-tls\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.125032 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.125002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd54k\" (UniqueName: \"kubernetes.io/projected/3f1dd432-d58b-4ea5-91a8-5fd041f6ad54-kube-api-access-fd54k\") pod \"metrics-server-7bd6857454-wqt6s\" (UID: \"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54\") " pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.242535 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.242490    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:29:58.279294 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.279263    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f595bfff8-lzmnt"]
Jul 27 17:29:58.321058 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.321023    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cb689474d-6hs6g"]
Jul 27 17:29:58.342365 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.342331    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb689474d-6hs6g"]
Jul 27 17:29:58.342522 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.342457    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.415818 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.415779    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-oauth-config\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.415992 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.415825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-trusted-ca-bundle\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.415992 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.415864    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-serving-cert\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.415992 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.415889    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-config\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.415992 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.415919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-service-ca\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.415992 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.415947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-oauth-serving-cert\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.415992 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.415973    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz84w\" (UniqueName: \"kubernetes.io/projected/ef5cdc13-4132-4804-bd5e-a5d068897bdf-kube-api-access-hz84w\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.517139 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.517056    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-oauth-config\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.517139 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.517098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-trusted-ca-bundle\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.517139 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.517122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-serving-cert\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.517450 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.517147    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-config\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.517450 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.517200    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-service-ca\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.517450 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.517227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-oauth-serving-cert\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.517450 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.517250    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz84w\" (UniqueName: \"kubernetes.io/projected/ef5cdc13-4132-4804-bd5e-a5d068897bdf-kube-api-access-hz84w\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.518442 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.518362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-oauth-serving-cert\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.518442 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.518370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-config\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.518442 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.518435    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-trusted-ca-bundle\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.518664 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.518635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-service-ca\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.520299 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.520274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-serving-cert\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.520434 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.520411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-oauth-config\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.528240 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.528159    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz84w\" (UniqueName: \"kubernetes.io/projected/ef5cdc13-4132-4804-bd5e-a5d068897bdf-kube-api-access-hz84w\") pod \"console-cb689474d-6hs6g\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") " pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.653595 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.653557    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:29:58.741882 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.741848    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"]
Jul 27 17:29:58.770454 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.770364    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"]
Jul 27 17:29:58.770613 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.770503    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:58.779152 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.779125    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 27 17:29:58.779300 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.779257    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 27 17:29:58.779507 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.779485    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 27 17:29:58.779507 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.779501    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 27 17:29:58.779684 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.779523    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 27 17:29:58.779738 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.779704    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n7kl9\""
Jul 27 17:29:58.790047 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.790025    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 27 17:29:58.921360 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.921329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-federate-client-tls\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:58.921360 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.921365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-secret-telemeter-client\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:58.921825 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.921410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-telemeter-client-tls\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:58.921825 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.921471    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57631304-d550-49a7-8e57-fddbecf9e6de-serving-certs-ca-bundle\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:58.921825 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.921530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfrx\" (UniqueName: \"kubernetes.io/projected/57631304-d550-49a7-8e57-fddbecf9e6de-kube-api-access-jmfrx\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:58.921825 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.921557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57631304-d550-49a7-8e57-fddbecf9e6de-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:58.921825 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.921584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:58.921825 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:58.921604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57631304-d550-49a7-8e57-fddbecf9e6de-metrics-client-ca\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.022613 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.022540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfrx\" (UniqueName: \"kubernetes.io/projected/57631304-d550-49a7-8e57-fddbecf9e6de-kube-api-access-jmfrx\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.022613 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.022581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57631304-d550-49a7-8e57-fddbecf9e6de-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.022613 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.022610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.022909 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.022643    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57631304-d550-49a7-8e57-fddbecf9e6de-metrics-client-ca\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.022909 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.022706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-federate-client-tls\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.022909 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.022741    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-secret-telemeter-client\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.022909 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.022790    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-telemeter-client-tls\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.022909 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.022849    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57631304-d550-49a7-8e57-fddbecf9e6de-serving-certs-ca-bundle\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.023522 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.023454    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57631304-d550-49a7-8e57-fddbecf9e6de-metrics-client-ca\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.023655 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.023590    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57631304-d550-49a7-8e57-fddbecf9e6de-serving-certs-ca-bundle\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.023655 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.023615    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57631304-d550-49a7-8e57-fddbecf9e6de-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.026220 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.026168    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.026348 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.026251    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-federate-client-tls\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.026348 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.026265    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-telemeter-client-tls\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.026723 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.026698    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/57631304-d550-49a7-8e57-fddbecf9e6de-secret-telemeter-client\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.052588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.052565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfrx\" (UniqueName: \"kubernetes.io/projected/57631304-d550-49a7-8e57-fddbecf9e6de-kube-api-access-jmfrx\") pod \"telemeter-client-5b6f69bf7c-w55bl\" (UID: \"57631304-d550-49a7-8e57-fddbecf9e6de\") " pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.082659 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.082620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"
Jul 27 17:29:59.816980 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.816948    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:29:59.859285 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.859238    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.862731 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.862640    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:29:59.864819 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.864580    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 17:29:59.864819 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.864668    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 17:29:59.864819 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.864730    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 17:29:59.864819 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.864580    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 17:29:59.864819 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.864751    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 17:29:59.865176 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.864846    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 17:29:59.865176 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.864860    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 17:29:59.865176 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.865121    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 17:29:59.865370 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.865238    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cm66b9h8m01hv\""
Jul 27 17:29:59.865370 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.865248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 17:29:59.865463 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.865444    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 17:29:59.865908 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.865888    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 17:29:59.874639 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.874622    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 17:29:59.874736 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.874670    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 17:29:59.875525 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.875415    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mwzr8\""
Jul 27 17:29:59.875652 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.875635    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 17:29:59.931494 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-config\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931494 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnk8j\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-kube-api-access-cnk8j\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931522    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931583    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931673    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931745    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931913    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.931923 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.932335 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931950    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.932335 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-web-config\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.932335 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.931990    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.932335 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.932009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-config-out\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.932335 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.932024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.932335 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:29:59.932069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:29:59.980352 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:29:59.980289    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a09cd19_3146_4617_961e_614b1edf601b.slice/crio-7dd2510972cb4b0446d7f880c2cbdc809ea39bc3c5d197627a5a382d167adbd6 WatchSource:0}: Error finding container 7dd2510972cb4b0446d7f880c2cbdc809ea39bc3c5d197627a5a382d167adbd6: Status 404 returned error can't find the container with id 7dd2510972cb4b0446d7f880c2cbdc809ea39bc3c5d197627a5a382d167adbd6
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.032761    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.032836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.032871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.032906    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.032936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-web-config\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.032964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.032993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-config-out\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.033020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.033086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.033283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.033159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-config\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.033964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnk8j\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-kube-api-access-cnk8j\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.034021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.034051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.034079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.034110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.034135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.034162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.034208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.034251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.034740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.034288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.036734 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.036283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.036734 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.036386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.038442 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.037391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.038442 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.038090    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.039202 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.039146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.039888 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.039811    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.047718 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.047329    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.049121 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.048369    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.049121 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.049013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.060988 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.060014    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.065270 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.062657    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-config-out\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.065270 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.062752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.065270 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.062855    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.065270 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.063286    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-config\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.065270 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.063643    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.065270 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.063826    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.069532 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.069380    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.071423 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.071374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnk8j\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-kube-api-access-cnk8j\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.072985 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.072937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.074324 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.073765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-web-config\") pod \"prometheus-k8s-0\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.149391 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.149364    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz"]
Jul 27 17:30:00.151350 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:30:00.151316    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4587b5df_0686_4ef4_b0e7_829add7246f4.slice/crio-bddc0609cdef40524cb2cd0b9bcc298e8f48248e0eb19149996b7320f82402c8 WatchSource:0}: Error finding container bddc0609cdef40524cb2cd0b9bcc298e8f48248e0eb19149996b7320f82402c8: Status 404 returned error can't find the container with id bddc0609cdef40524cb2cd0b9bcc298e8f48248e0eb19149996b7320f82402c8
Jul 27 17:30:00.163103 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.163027    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl"]
Jul 27 17:30:00.165591 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:30:00.165566    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57631304_d550_49a7_8e57_fddbecf9e6de.slice/crio-a3fefdac178bb3971d62dc3ca6537781fcb07081532ce3278e1a4dabecd83b13 WatchSource:0}: Error finding container a3fefdac178bb3971d62dc3ca6537781fcb07081532ce3278e1a4dabecd83b13: Status 404 returned error can't find the container with id a3fefdac178bb3971d62dc3ca6537781fcb07081532ce3278e1a4dabecd83b13
Jul 27 17:30:00.171383 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.171358    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:00.383336 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.383306    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb689474d-6hs6g"]
Jul 27 17:30:00.385610 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:30:00.385554    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5cdc13_4132_4804_bd5e_a5d068897bdf.slice/crio-33030b44a5e80b9b57ea01dfc21f0525a273b957cf239c3728360e21cd36b2d1 WatchSource:0}: Error finding container 33030b44a5e80b9b57ea01dfc21f0525a273b957cf239c3728360e21cd36b2d1: Status 404 returned error can't find the container with id 33030b44a5e80b9b57ea01dfc21f0525a273b957cf239c3728360e21cd36b2d1
Jul 27 17:30:00.387314 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.387291    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bd6857454-wqt6s"]
Jul 27 17:30:00.389751 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:30:00.389718    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1dd432_d58b_4ea5_91a8_5fd041f6ad54.slice/crio-555fbfb948f5c60cf8530bc455c4a9fd85b932478219b322b22a430dba6e5589 WatchSource:0}: Error finding container 555fbfb948f5c60cf8530bc455c4a9fd85b932478219b322b22a430dba6e5589: Status 404 returned error can't find the container with id 555fbfb948f5c60cf8530bc455c4a9fd85b932478219b322b22a430dba6e5589
Jul 27 17:30:00.498055 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.498024    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:30:00.501754 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:30:00.501717    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920a00fa_eb6f_46e5_a08a_e073e31a471f.slice/crio-6a5eb4eab08fce73ad6983410db502e4bc2835298e8014ba878c8992dcf1a0d5 WatchSource:0}: Error finding container 6a5eb4eab08fce73ad6983410db502e4bc2835298e8014ba878c8992dcf1a0d5: Status 404 returned error can't find the container with id 6a5eb4eab08fce73ad6983410db502e4bc2835298e8014ba878c8992dcf1a0d5
Jul 27 17:30:00.707225 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.707053    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb689474d-6hs6g" event={"ID":"ef5cdc13-4132-4804-bd5e-a5d068897bdf","Type":"ContainerStarted","Data":"b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a"}
Jul 27 17:30:00.707225 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.707116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb689474d-6hs6g" event={"ID":"ef5cdc13-4132-4804-bd5e-a5d068897bdf","Type":"ContainerStarted","Data":"33030b44a5e80b9b57ea01dfc21f0525a273b957cf239c3728360e21cd36b2d1"}
Jul 27 17:30:00.709282 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.709222    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz" event={"ID":"4587b5df-0686-4ef4-b0e7-829add7246f4","Type":"ContainerStarted","Data":"bddc0609cdef40524cb2cd0b9bcc298e8f48248e0eb19149996b7320f82402c8"}
Jul 27 17:30:00.710601 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.710525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerStarted","Data":"6a5eb4eab08fce73ad6983410db502e4bc2835298e8014ba878c8992dcf1a0d5"}
Jul 27 17:30:00.713554 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.713115    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-nlscd" event={"ID":"ee681144-a38e-4964-9c30-604dfd4e6d71","Type":"ContainerStarted","Data":"3ddf1d184e7f3039fd7cb6348b7ff10b015b155b0a65d1ea370d7a5582368f9a"}
Jul 27 17:30:00.713554 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.713297    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-nlscd"
Jul 27 17:30:00.715084 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.714586    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-629cq" event={"ID":"1a09cd19-3146-4617-961e-614b1edf601b","Type":"ContainerStarted","Data":"7dd2510972cb4b0446d7f880c2cbdc809ea39bc3c5d197627a5a382d167adbd6"}
Jul 27 17:30:00.716472 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.716436    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl" event={"ID":"57631304-d550-49a7-8e57-fddbecf9e6de","Type":"ContainerStarted","Data":"a3fefdac178bb3971d62dc3ca6537781fcb07081532ce3278e1a4dabecd83b13"}
Jul 27 17:30:00.717776 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.717724    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s" event={"ID":"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54","Type":"ContainerStarted","Data":"555fbfb948f5c60cf8530bc455c4a9fd85b932478219b322b22a430dba6e5589"}
Jul 27 17:30:00.726531 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.726411    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-nlscd"
Jul 27 17:30:00.727310 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.727262    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cb689474d-6hs6g" podStartSLOduration=2.727243965 podStartE2EDuration="2.727243965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:00.726266326 +0000 UTC m=+144.178887474" watchObservedRunningTime="2026-07-27 17:30:00.727243965 +0000 UTC m=+144.179865114"
Jul 27 17:30:00.747729 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:00.747035    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-nlscd" podStartSLOduration=1.199996843 podStartE2EDuration="21.747015172s" totalImagesPullingTime="20.547018329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:00.74435053 +0000 UTC m=+144.196971699" watchObservedRunningTime="2026-07-27 17:30:00.747015172 +0000 UTC m=+144.199636321"
Jul 27 17:30:01.730214 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:01.729775    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-629cq" event={"ID":"1a09cd19-3146-4617-961e-614b1edf601b","Type":"ContainerStarted","Data":"a760165a8d13488f43e146ad6fbc7a5012e68283b0bd4587f58f432b643441fd"}
Jul 27 17:30:02.735630 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:02.735592    2564 generic.go:356] "Generic (PLEG): container finished" podID="1a09cd19-3146-4617-961e-614b1edf601b" containerID="a760165a8d13488f43e146ad6fbc7a5012e68283b0bd4587f58f432b643441fd" exitCode=0
Jul 27 17:30:02.736058 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:02.735681    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-629cq" event={"ID":"1a09cd19-3146-4617-961e-614b1edf601b","Type":"ContainerDied","Data":"a760165a8d13488f43e146ad6fbc7a5012e68283b0bd4587f58f432b643441fd"}
Jul 27 17:30:04.747044 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:04.747003    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s" event={"ID":"3f1dd432-d58b-4ea5-91a8-5fd041f6ad54","Type":"ContainerStarted","Data":"a4e310f87cb77552791dd5a5a19589810bf947c9238dba37caabd9e0423bceda"}
Jul 27 17:30:04.748938 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:04.748873    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz" event={"ID":"4587b5df-0686-4ef4-b0e7-829add7246f4","Type":"ContainerStarted","Data":"e1f39e28fbe003b9f2a2cfbd64f134bb9f98e668cbd218e0bb850ead40d3f9c4"}
Jul 27 17:30:04.752097 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:04.752073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerStarted","Data":"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"}
Jul 27 17:30:04.754530 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:04.754391    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-629cq" event={"ID":"1a09cd19-3146-4617-961e-614b1edf601b","Type":"ContainerStarted","Data":"00f23d78cd016dd2524f38543a91a8a86ac00ec950ada50c62fb565a34691095"}
Jul 27 17:30:04.786906 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:04.784828    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s" podStartSLOduration=3.596791609 podStartE2EDuration="7.784806946s" totalImagesPullingTime="4.188015337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:04.782429101 +0000 UTC m=+148.235050242" watchObservedRunningTime="2026-07-27 17:30:04.784806946 +0000 UTC m=+148.237428094"
Jul 27 17:30:05.761513 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.761466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-629cq" event={"ID":"1a09cd19-3146-4617-961e-614b1edf601b","Type":"ContainerStarted","Data":"ab8be2143e75027ab6c09f0295bf00f0f8cd83692bc6c22ba37c2a5d25d90f93"}
Jul 27 17:30:05.763940 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.763911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl" event={"ID":"57631304-d550-49a7-8e57-fddbecf9e6de","Type":"ContainerStarted","Data":"c65e1f152eb89d7dac0b49a3079553e808b1d91b62dd320412f6d488724c09b9"}
Jul 27 17:30:05.764147 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.764120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl" event={"ID":"57631304-d550-49a7-8e57-fddbecf9e6de","Type":"ContainerStarted","Data":"ab6b2207e3787d1d6fb8d29af7eae560ed6c11410d96f7d2a6e5a69d1ee8965d"}
Jul 27 17:30:05.764280 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.764159    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl" event={"ID":"57631304-d550-49a7-8e57-fddbecf9e6de","Type":"ContainerStarted","Data":"affee03483919aaf3b797f6dcdb2e6503f1a14e12368623231fd3ab7f853e511"}
Jul 27 17:30:05.766406 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.766378    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz" event={"ID":"4587b5df-0686-4ef4-b0e7-829add7246f4","Type":"ContainerStarted","Data":"b05754b70c97b276e77b5601ebe605bd9bd35cf5623b2298634f0dae3d5fbd1c"}
Jul 27 17:30:05.766509 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.766436    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz" event={"ID":"4587b5df-0686-4ef4-b0e7-829add7246f4","Type":"ContainerStarted","Data":"7e50e705d7be84d738e7aeec624561fd9e36bf42a0f69416a62049e978d268f6"}
Jul 27 17:30:05.771071 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.771042    2564 generic.go:356] "Generic (PLEG): container finished" podID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerID="979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49" exitCode=0
Jul 27 17:30:05.771212 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.771131    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerDied","Data":"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"}
Jul 27 17:30:05.785720 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.785374    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-629cq" podStartSLOduration=11.336660858 podStartE2EDuration="12.785353495s" totalImagesPullingTime="1.330814908s" totalInitContainerRuntime="117.877729ms" isStatefulPod=true podCreationTimestamp="2026-07-27 17:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:05.784505777 +0000 UTC m=+149.237126923" watchObservedRunningTime="2026-07-27 17:30:05.785353495 +0000 UTC m=+149.237974643"
Jul 27 17:30:05.809693 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.809639    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-45xpz" podStartSLOduration=8.383739229 podStartE2EDuration="12.809619477s" totalImagesPullingTime="4.425880248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:05.807657858 +0000 UTC m=+149.260279005" watchObservedRunningTime="2026-07-27 17:30:05.809619477 +0000 UTC m=+149.262240627"
Jul 27 17:30:05.842895 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.842827    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b6f69bf7c-w55bl" podStartSLOduration=3.424479188 podStartE2EDuration="7.842806844s" totalImagesPullingTime="4.418327656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:05.83991317 +0000 UTC m=+149.292534329" watchObservedRunningTime="2026-07-27 17:30:05.842806844 +0000 UTC m=+149.295427989"
Jul 27 17:30:05.951193 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:05.951157    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b6d68669d-jrdfl"]
Jul 27 17:30:08.653803 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:08.653762    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:30:08.654310 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:08.653838    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:30:08.655352 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:08.655325    2564 patch_prober.go:28] interesting pod/console-cb689474d-6hs6g container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body=
Jul 27 17:30:08.655472 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:08.655378    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-cb689474d-6hs6g" podUID="ef5cdc13-4132-4804-bd5e-a5d068897bdf" containerName="console" probeResult="failure" output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused"
Jul 27 17:30:13.919564 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:30:13.919510    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx" podUID="9abb877f-c36e-4524-bf95-241c606335ce"
Jul 27 17:30:13.931712 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:30:13.931669    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xt257" podUID="332d404e-0e71-485a-8a0d-05951cf4752d"
Jul 27 17:30:14.808865 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:14.808828    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:30:14.809066 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:14.808828    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xt257"
Jul 27 17:30:16.819784 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:16.819751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerStarted","Data":"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"}
Jul 27 17:30:16.820258 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:16.819820    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerStarted","Data":"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"}
Jul 27 17:30:18.242717 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.242691    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:30:18.243012 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.242768    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:30:18.653935 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.653894    2564 patch_prober.go:28] interesting pod/console-cb689474d-6hs6g container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body=
Jul 27 17:30:18.654157 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.653943    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-cb689474d-6hs6g" podUID="ef5cdc13-4132-4804-bd5e-a5d068897bdf" containerName="console" probeResult="failure" output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused"
Jul 27 17:30:18.831013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.830917    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerStarted","Data":"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"}
Jul 27 17:30:18.831013 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.831002    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerStarted","Data":"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"}
Jul 27 17:30:18.831271 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.831047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerStarted","Data":"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"}
Jul 27 17:30:18.831271 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.831090    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerStarted","Data":"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"}
Jul 27 17:30:18.864167 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.864109    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.523569202 podStartE2EDuration="19.864090163s" totalImagesPullingTime="16.294908768s" totalInitContainerRuntime="45.612193ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:18.862084431 +0000 UTC m=+162.314705588" watchObservedRunningTime="2026-07-27 17:30:18.864090163 +0000 UTC m=+162.316711310"
Jul 27 17:30:18.931874 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.931833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:30:18.932250 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.932219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:30:18.934468 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.934442    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332d404e-0e71-485a-8a0d-05951cf4752d-metrics-tls\") pod \"dns-default-xt257\" (UID: \"332d404e-0e71-485a-8a0d-05951cf4752d\") " pod="openshift-dns/dns-default-xt257"
Jul 27 17:30:18.934900 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:18.934873    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"image-registry-666b8c6bcb-f2nbx\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") " pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:30:19.012539 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.012507    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-swd8n\""
Jul 27 17:30:19.012539 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.012530    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h7n9r\""
Jul 27 17:30:19.019896 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.019875    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:30:19.019992 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.019964    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xt257"
Jul 27 17:30:19.202245 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.202217    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666b8c6bcb-f2nbx"]
Jul 27 17:30:19.204445 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:30:19.204420    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abb877f_c36e_4524_bf95_241c606335ce.slice/crio-a39612e61f29c9573e56a1fdc5318609dcb722269b9a79dc52eb1b7adee66690 WatchSource:0}: Error finding container a39612e61f29c9573e56a1fdc5318609dcb722269b9a79dc52eb1b7adee66690: Status 404 returned error can't find the container with id a39612e61f29c9573e56a1fdc5318609dcb722269b9a79dc52eb1b7adee66690
Jul 27 17:30:19.232950 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.232925    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xt257"]
Jul 27 17:30:19.233635 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:30:19.233608    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332d404e_0e71_485a_8a0d_05951cf4752d.slice/crio-78081962c1d5a6ec2302163afe40f26b9016b2c2c898924a583df2e7da5cd756 WatchSource:0}: Error finding container 78081962c1d5a6ec2302163afe40f26b9016b2c2c898924a583df2e7da5cd756: Status 404 returned error can't find the container with id 78081962c1d5a6ec2302163afe40f26b9016b2c2c898924a583df2e7da5cd756
Jul 27 17:30:19.837736 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.837703    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx" event={"ID":"9abb877f-c36e-4524-bf95-241c606335ce","Type":"ContainerStarted","Data":"09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d"}
Jul 27 17:30:19.838281 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.837759    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx" event={"ID":"9abb877f-c36e-4524-bf95-241c606335ce","Type":"ContainerStarted","Data":"a39612e61f29c9573e56a1fdc5318609dcb722269b9a79dc52eb1b7adee66690"}
Jul 27 17:30:19.838281 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.837822    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:30:19.839214 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.839174    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xt257" event={"ID":"332d404e-0e71-485a-8a0d-05951cf4752d","Type":"ContainerStarted","Data":"78081962c1d5a6ec2302163afe40f26b9016b2c2c898924a583df2e7da5cd756"}
Jul 27 17:30:19.861539 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:19.861491    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx" podStartSLOduration=162.861473801 podStartE2EDuration="2m42.861473801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:19.860302743 +0000 UTC m=+163.312923893" watchObservedRunningTime="2026-07-27 17:30:19.861473801 +0000 UTC m=+163.314094948"
Jul 27 17:30:20.172225 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:20.172121    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:30:21.850323 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:21.850234    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xt257" event={"ID":"332d404e-0e71-485a-8a0d-05951cf4752d","Type":"ContainerStarted","Data":"0800e56db39deef218e00ac60370131f65f25730a44f58e2444de734a58d7f3f"}
Jul 27 17:30:21.850323 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:21.850301    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xt257" event={"ID":"332d404e-0e71-485a-8a0d-05951cf4752d","Type":"ContainerStarted","Data":"8479ff17c250e2c151957b40b067d42aa12bf88b75a88b0b6a7dddf9007597a6"}
Jul 27 17:30:21.850838 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:21.850333    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xt257"
Jul 27 17:30:21.870334 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:21.870286    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xt257" podStartSLOduration=129.817483219 podStartE2EDuration="2m11.870272051s" totalImagesPullingTime="2.052788832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:21.869137863 +0000 UTC m=+165.321759009" watchObservedRunningTime="2026-07-27 17:30:21.870272051 +0000 UTC m=+165.322893217"
Jul 27 17:30:23.300041 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:23.299996    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f595bfff8-lzmnt" podUID="7ecc8a0e-47c6-458f-8862-c787e6d59f1e" containerName="console" containerID="cri-o://63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397" gracePeriod=15
Jul 27 17:30:25.867362 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:25.867326    2564 generic.go:356] "Generic (PLEG): container finished" podID="9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989" containerID="61f964a58916aca1234fed32a1ddcf1ae51d58a728cd0a0e71e3a06038ffd5e1" exitCode=0
Jul 27 17:30:25.867749 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:25.867400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p" event={"ID":"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989","Type":"ContainerDied","Data":"61f964a58916aca1234fed32a1ddcf1ae51d58a728cd0a0e71e3a06038ffd5e1"}
Jul 27 17:30:25.867749 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:25.867705    2564 scope.go:117] "RemoveContainer" containerID="61f964a58916aca1234fed32a1ddcf1ae51d58a728cd0a0e71e3a06038ffd5e1"
Jul 27 17:30:26.873085 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:26.873033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zk57p" event={"ID":"9e27191f-4d7b-4bd5-a7b3-9e43cfbd4989","Type":"ContainerStarted","Data":"6bb3c6ecfad82e17c5cf6935631fd89256ac2c43368b1b74c6a12e94755fd1cb"}
Jul 27 17:30:28.654841 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:28.654756    2564 patch_prober.go:28] interesting pod/console-cb689474d-6hs6g container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body=
Jul 27 17:30:28.654841 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:28.654804    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-cb689474d-6hs6g" podUID="ef5cdc13-4132-4804-bd5e-a5d068897bdf" containerName="console" probeResult="failure" output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused"
Jul 27 17:30:30.975762 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:30.975720    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b6d68669d-jrdfl" podUID="c8b4dec9-4516-48fd-ad49-5f4456c8288d" containerName="console" containerID="cri-o://a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09" gracePeriod=15
Jul 27 17:30:31.856476 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:31.856449    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xt257"
Jul 27 17:30:36.165696 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.165668    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:30:36.297233 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.297177    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-oauth-config\") pod \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") "
Jul 27 17:30:36.297408 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.297246    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-service-ca\") pod \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") "
Jul 27 17:30:36.297408 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.297287    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-serving-cert\") pod \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") "
Jul 27 17:30:36.297408 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.297327    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c24mr\" (UniqueName: \"kubernetes.io/projected/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-kube-api-access-c24mr\") pod \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") "
Jul 27 17:30:36.297408 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.297369    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-oauth-serving-cert\") pod \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") "
Jul 27 17:30:36.297408 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.297397    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-config\") pod \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\" (UID: \"7ecc8a0e-47c6-458f-8862-c787e6d59f1e\") "
Jul 27 17:30:36.297749 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.297714    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-service-ca" (OuterVolumeSpecName: "service-ca") pod "7ecc8a0e-47c6-458f-8862-c787e6d59f1e" (UID: "7ecc8a0e-47c6-458f-8862-c787e6d59f1e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:30:36.297880 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.297806    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-config" (OuterVolumeSpecName: "console-config") pod "7ecc8a0e-47c6-458f-8862-c787e6d59f1e" (UID: "7ecc8a0e-47c6-458f-8862-c787e6d59f1e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:30:36.297880 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.297817    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7ecc8a0e-47c6-458f-8862-c787e6d59f1e" (UID: "7ecc8a0e-47c6-458f-8862-c787e6d59f1e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:30:36.299532 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.299507    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7ecc8a0e-47c6-458f-8862-c787e6d59f1e" (UID: "7ecc8a0e-47c6-458f-8862-c787e6d59f1e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:36.299641 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.299598    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7ecc8a0e-47c6-458f-8862-c787e6d59f1e" (UID: "7ecc8a0e-47c6-458f-8862-c787e6d59f1e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:36.299701 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.299643    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-kube-api-access-c24mr" (OuterVolumeSpecName: "kube-api-access-c24mr") pod "7ecc8a0e-47c6-458f-8862-c787e6d59f1e" (UID: "7ecc8a0e-47c6-458f-8862-c787e6d59f1e"). InnerVolumeSpecName "kube-api-access-c24mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:30:36.370823 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.370785    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb689474d-6hs6g"]
Jul 27 17:30:36.398203 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.398149    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-oauth-serving-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.398203 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.398175    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.398203 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.398209    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-oauth-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.398425 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.398218    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-service-ca\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.398425 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.398227    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-console-serving-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.398425 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.398234    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c24mr\" (UniqueName: \"kubernetes.io/projected/7ecc8a0e-47c6-458f-8862-c787e6d59f1e-kube-api-access-c24mr\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.631272 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.631242    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:30:36.801195 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801149    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-oauth-config\") pod \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") "
Jul 27 17:30:36.801379 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801209    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-trusted-ca-bundle\") pod \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") "
Jul 27 17:30:36.801379 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801230    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-oauth-serving-cert\") pod \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") "
Jul 27 17:30:36.801379 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801249    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-config\") pod \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") "
Jul 27 17:30:36.801379 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801278    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-serving-cert\") pod \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") "
Jul 27 17:30:36.801628 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801374    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjbtp\" (UniqueName: \"kubernetes.io/projected/c8b4dec9-4516-48fd-ad49-5f4456c8288d-kube-api-access-fjbtp\") pod \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") "
Jul 27 17:30:36.801628 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801453    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-service-ca\") pod \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\" (UID: \"c8b4dec9-4516-48fd-ad49-5f4456c8288d\") "
Jul 27 17:30:36.801740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801666    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c8b4dec9-4516-48fd-ad49-5f4456c8288d" (UID: "c8b4dec9-4516-48fd-ad49-5f4456c8288d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:30:36.801740 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801688    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-config" (OuterVolumeSpecName: "console-config") pod "c8b4dec9-4516-48fd-ad49-5f4456c8288d" (UID: "c8b4dec9-4516-48fd-ad49-5f4456c8288d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:30:36.801851 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801821    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-oauth-serving-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.801851 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801711    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c8b4dec9-4516-48fd-ad49-5f4456c8288d" (UID: "c8b4dec9-4516-48fd-ad49-5f4456c8288d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:30:36.801936 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801839    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.801975 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.801919    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-service-ca" (OuterVolumeSpecName: "service-ca") pod "c8b4dec9-4516-48fd-ad49-5f4456c8288d" (UID: "c8b4dec9-4516-48fd-ad49-5f4456c8288d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:30:36.803416 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.803389    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c8b4dec9-4516-48fd-ad49-5f4456c8288d" (UID: "c8b4dec9-4516-48fd-ad49-5f4456c8288d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:36.803529 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.803497    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b4dec9-4516-48fd-ad49-5f4456c8288d-kube-api-access-fjbtp" (OuterVolumeSpecName: "kube-api-access-fjbtp") pod "c8b4dec9-4516-48fd-ad49-5f4456c8288d" (UID: "c8b4dec9-4516-48fd-ad49-5f4456c8288d"). InnerVolumeSpecName "kube-api-access-fjbtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:30:36.803592 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.803578    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c8b4dec9-4516-48fd-ad49-5f4456c8288d" (UID: "c8b4dec9-4516-48fd-ad49-5f4456c8288d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:36.902855 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.902822    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-oauth-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.902855 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.902849    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-trusted-ca-bundle\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.902855 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.902859    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8b4dec9-4516-48fd-ad49-5f4456c8288d-console-serving-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.903082 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.902868    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbtp\" (UniqueName: \"kubernetes.io/projected/c8b4dec9-4516-48fd-ad49-5f4456c8288d-kube-api-access-fjbtp\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.903082 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.902879    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b4dec9-4516-48fd-ad49-5f4456c8288d-service-ca\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:30:36.917774 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.917737    2564 generic.go:356] "Generic (PLEG): container finished" podID="7ecc8a0e-47c6-458f-8862-c787e6d59f1e" containerID="63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397" exitCode=0
Jul 27 17:30:36.917774 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.917770    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f595bfff8-lzmnt"
Jul 27 17:30:36.917976 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.917815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f595bfff8-lzmnt" event={"ID":"7ecc8a0e-47c6-458f-8862-c787e6d59f1e","Type":"ContainerDied","Data":"63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397"}
Jul 27 17:30:36.917976 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.917862    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f595bfff8-lzmnt" event={"ID":"7ecc8a0e-47c6-458f-8862-c787e6d59f1e","Type":"ContainerDied","Data":"a341b23916179438c0f3289baca9836adb1b61fd7226996add566e2016b1f8ac"}
Jul 27 17:30:36.917976 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.917882    2564 scope.go:117] "RemoveContainer" containerID="63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397"
Jul 27 17:30:36.919303 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.919239    2564 generic.go:356] "Generic (PLEG): container finished" podID="c8b4dec9-4516-48fd-ad49-5f4456c8288d" containerID="a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09" exitCode=0
Jul 27 17:30:36.919303 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.919301    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6d68669d-jrdfl"
Jul 27 17:30:36.919468 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.919330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6d68669d-jrdfl" event={"ID":"c8b4dec9-4516-48fd-ad49-5f4456c8288d","Type":"ContainerDied","Data":"a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09"}
Jul 27 17:30:36.919468 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.919437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6d68669d-jrdfl" event={"ID":"c8b4dec9-4516-48fd-ad49-5f4456c8288d","Type":"ContainerDied","Data":"15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4"}
Jul 27 17:30:36.927382 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.927365    2564 scope.go:117] "RemoveContainer" containerID="63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397"
Jul 27 17:30:36.927708 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:30:36.927678    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397\": container with ID starting with 63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397 not found: ID does not exist" containerID="63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397"
Jul 27 17:30:36.927801 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.927719    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397"} err="failed to get container status \"63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397\": rpc error: code = NotFound desc = could not find container \"63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397\": container with ID starting with 63131fc8e8238808242956cef789cfad5fcb867dea7870b6065ab3beeb762397 not found: ID does not exist"
Jul 27 17:30:36.927801 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.927760    2564 scope.go:117] "RemoveContainer" containerID="a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09"
Jul 27 17:30:36.936248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.936234    2564 scope.go:117] "RemoveContainer" containerID="a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09"
Jul 27 17:30:36.936511 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:30:36.936488    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09\": container with ID starting with a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09 not found: ID does not exist" containerID="a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09"
Jul 27 17:30:36.936605 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.936514    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09"} err="failed to get container status \"a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09\": rpc error: code = NotFound desc = could not find container \"a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09\": container with ID starting with a311226ceb83bb6ba10b329a70ed71dec53e7cb09742a14ca6b780b8e76c7d09 not found: ID does not exist"
Jul 27 17:30:36.945137 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.945109    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f595bfff8-lzmnt"]
Jul 27 17:30:36.951100 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.951080    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f595bfff8-lzmnt"]
Jul 27 17:30:36.961816 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.961795    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b6d68669d-jrdfl"]
Jul 27 17:30:36.964558 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:36.964538    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b6d68669d-jrdfl"]
Jul 27 17:30:37.120400 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:37.120325    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ecc8a0e-47c6-458f-8862-c787e6d59f1e" path="/var/lib/kubelet/pods/7ecc8a0e-47c6-458f-8862-c787e6d59f1e/volumes"
Jul 27 17:30:37.120726 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:37.120712    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8b4dec9-4516-48fd-ad49-5f4456c8288d" path="/var/lib/kubelet/pods/c8b4dec9-4516-48fd-ad49-5f4456c8288d/volumes"
Jul 27 17:30:37.351319 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:30:37.351280    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 17:30:38.248016 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:38.247987    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:30:38.251785 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:38.251756    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bd6857454-wqt6s"
Jul 27 17:30:38.811961 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:38.811920    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-45xpz_4587b5df-0686-4ef4-b0e7-829add7246f4/kube-state-metrics/0.log"
Jul 27 17:30:39.011914 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:39.011885    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-45xpz_4587b5df-0686-4ef4-b0e7-829add7246f4/kube-rbac-proxy-main/0.log"
Jul 27 17:30:39.024960 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:39.024932    2564 patch_prober.go:28] interesting pod/image-registry-666b8c6bcb-f2nbx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:30:39.025097 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:39.024992    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx" podUID="9abb877f-c36e-4524-bf95-241c606335ce" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:30:39.211857 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:39.211778    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-45xpz_4587b5df-0686-4ef4-b0e7-829add7246f4/kube-rbac-proxy-self/0.log"
Jul 27 17:30:39.410824 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:39.410794    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bd6857454-wqt6s_3f1dd432-d58b-4ea5-91a8-5fd041f6ad54/metrics-server/0.log"
Jul 27 17:30:39.811083 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:39.811058    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-629cq_1a09cd19-3146-4617-961e-614b1edf601b/init-textfile/0.log"
Jul 27 17:30:40.014591 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:40.014557    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-629cq_1a09cd19-3146-4617-961e-614b1edf601b/node-exporter/0.log"
Jul 27 17:30:40.214039 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:40.213969    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-629cq_1a09cd19-3146-4617-961e-614b1edf601b/kube-rbac-proxy/0.log"
Jul 27 17:30:40.849133 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:40.849104    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:30:42.211218 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:42.211169    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_920a00fa-eb6f-46e5-a08a-e073e31a471f/init-config-reloader/0.log"
Jul 27 17:30:42.412313 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:42.412260    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_920a00fa-eb6f-46e5-a08a-e073e31a471f/prometheus/0.log"
Jul 27 17:30:42.615039 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:42.615011    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_920a00fa-eb6f-46e5-a08a-e073e31a471f/config-reloader/0.log"
Jul 27 17:30:42.811354 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:42.811325    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_920a00fa-eb6f-46e5-a08a-e073e31a471f/thanos-sidecar/0.log"
Jul 27 17:30:43.011322 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:43.011295    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_920a00fa-eb6f-46e5-a08a-e073e31a471f/kube-rbac-proxy-web/0.log"
Jul 27 17:30:43.210646 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:43.210622    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_920a00fa-eb6f-46e5-a08a-e073e31a471f/kube-rbac-proxy/0.log"
Jul 27 17:30:43.411595 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:43.411504    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_920a00fa-eb6f-46e5-a08a-e073e31a471f/kube-rbac-proxy-thanos/0.log"
Jul 27 17:30:44.211299 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:44.211270    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b6f69bf7c-w55bl_57631304-d550-49a7-8e57-fddbecf9e6de/telemeter-client/0.log"
Jul 27 17:30:44.410935 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:44.410907    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b6f69bf7c-w55bl_57631304-d550-49a7-8e57-fddbecf9e6de/reload/0.log"
Jul 27 17:30:44.613158 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:44.613128    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b6f69bf7c-w55bl_57631304-d550-49a7-8e57-fddbecf9e6de/kube-rbac-proxy/0.log"
Jul 27 17:30:46.010957 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:46.010925    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-865bb_e73495df-4bcc-4b05-883e-a539562031b4/networking-console-plugin/0.log"
Jul 27 17:30:46.212254 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:46.212229    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:30:46.414961 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:46.414885    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/2.log"
Jul 27 17:30:47.212160 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:47.212129    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-cb689474d-6hs6g_ef5cdc13-4132-4804-bd5e-a5d068897bdf/console/0.log"
Jul 27 17:30:47.392982 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:30:47.392939    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 17:30:47.415732 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:47.415700    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-nlscd_ee681144-a38e-4964-9c30-604dfd4e6d71/download-server/0.log"
Jul 27 17:30:48.011588 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:48.011561    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xt257_332d404e-0e71-485a-8a0d-05951cf4752d/dns/0.log"
Jul 27 17:30:48.210912 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:48.210882    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xt257_332d404e-0e71-485a-8a0d-05951cf4752d/kube-rbac-proxy/0.log"
Jul 27 17:30:48.811095 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:48.811067    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2lkcp_df9a0647-8600-491a-8042-186017147267/dns-node-resolver/0.log"
Jul 27 17:30:49.411932 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:49.411876    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-666b8c6bcb-f2nbx_9abb877f-c36e-4524-bf95-241c606335ce/registry/0.log"
Jul 27 17:30:50.011581 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:30:50.011552    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v8rh6_9c64d459-de22-4e66-a489-4c7ac0a62897/node-ca/0.log"
Jul 27 17:30:50.596773 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:30:50.596732    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache]"
Jul 27 17:30:57.435359 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:30:57.432348    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:00.171839 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:00.171803    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:00.191525 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:00.191501    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:01.035972 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.035946    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:01.389908 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.389806    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cb689474d-6hs6g" podUID="ef5cdc13-4132-4804-bd5e-a5d068897bdf" containerName="console" containerID="cri-o://b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a" gracePeriod=15
Jul 27 17:31:01.641537 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.641477    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:31:01.723620 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.723590    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-service-ca\") pod \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") "
Jul 27 17:31:01.723620 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.723621    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-config\") pod \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") "
Jul 27 17:31:01.723816 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.723669    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-oauth-config\") pod \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") "
Jul 27 17:31:01.723816 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.723740    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-serving-cert\") pod \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") "
Jul 27 17:31:01.723893 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.723869    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-oauth-serving-cert\") pod \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") "
Jul 27 17:31:01.723933 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.723915    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-trusted-ca-bundle\") pod \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") "
Jul 27 17:31:01.723984 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.723948    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz84w\" (UniqueName: \"kubernetes.io/projected/ef5cdc13-4132-4804-bd5e-a5d068897bdf-kube-api-access-hz84w\") pod \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\" (UID: \"ef5cdc13-4132-4804-bd5e-a5d068897bdf\") "
Jul 27 17:31:01.724036 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.724013    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-config" (OuterVolumeSpecName: "console-config") pod "ef5cdc13-4132-4804-bd5e-a5d068897bdf" (UID: "ef5cdc13-4132-4804-bd5e-a5d068897bdf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:01.724264 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.724245    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:01.724426 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.724265    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ef5cdc13-4132-4804-bd5e-a5d068897bdf" (UID: "ef5cdc13-4132-4804-bd5e-a5d068897bdf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:01.724555 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.724392    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-service-ca" (OuterVolumeSpecName: "service-ca") pod "ef5cdc13-4132-4804-bd5e-a5d068897bdf" (UID: "ef5cdc13-4132-4804-bd5e-a5d068897bdf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:01.724555 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.724435    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ef5cdc13-4132-4804-bd5e-a5d068897bdf" (UID: "ef5cdc13-4132-4804-bd5e-a5d068897bdf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:01.725873 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.725843    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ef5cdc13-4132-4804-bd5e-a5d068897bdf" (UID: "ef5cdc13-4132-4804-bd5e-a5d068897bdf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:01.726312 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.726274    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ef5cdc13-4132-4804-bd5e-a5d068897bdf" (UID: "ef5cdc13-4132-4804-bd5e-a5d068897bdf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:01.726437 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.726416    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5cdc13-4132-4804-bd5e-a5d068897bdf-kube-api-access-hz84w" (OuterVolumeSpecName: "kube-api-access-hz84w") pod "ef5cdc13-4132-4804-bd5e-a5d068897bdf" (UID: "ef5cdc13-4132-4804-bd5e-a5d068897bdf"). InnerVolumeSpecName "kube-api-access-hz84w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:01.825758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.825726    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-service-ca\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:01.825758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.825752    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-oauth-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:01.825758 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.825761    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5cdc13-4132-4804-bd5e-a5d068897bdf-console-serving-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:01.825971 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.825770    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-oauth-serving-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:01.825971 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.825779    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef5cdc13-4132-4804-bd5e-a5d068897bdf-trusted-ca-bundle\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:01.825971 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:01.825788    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz84w\" (UniqueName: \"kubernetes.io/projected/ef5cdc13-4132-4804-bd5e-a5d068897bdf-kube-api-access-hz84w\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:02.030835 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:02.030801    2564 generic.go:356] "Generic (PLEG): container finished" podID="ef5cdc13-4132-4804-bd5e-a5d068897bdf" containerID="b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a" exitCode=0
Jul 27 17:31:02.031037 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:02.030866    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb689474d-6hs6g"
Jul 27 17:31:02.031037 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:02.030890    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb689474d-6hs6g" event={"ID":"ef5cdc13-4132-4804-bd5e-a5d068897bdf","Type":"ContainerDied","Data":"b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a"}
Jul 27 17:31:02.031037 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:02.030984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb689474d-6hs6g" event={"ID":"ef5cdc13-4132-4804-bd5e-a5d068897bdf","Type":"ContainerDied","Data":"33030b44a5e80b9b57ea01dfc21f0525a273b957cf239c3728360e21cd36b2d1"}
Jul 27 17:31:02.031037 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:02.031011    2564 scope.go:117] "RemoveContainer" containerID="b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a"
Jul 27 17:31:02.042319 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:02.042278    2564 scope.go:117] "RemoveContainer" containerID="b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a"
Jul 27 17:31:02.042700 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:02.042680    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a\": container with ID starting with b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a not found: ID does not exist" containerID="b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a"
Jul 27 17:31:02.042762 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:02.042708    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a"} err="failed to get container status \"b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a\": rpc error: code = NotFound desc = could not find container \"b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a\": container with ID starting with b810677a6156e03c2bebe0553b6c5d8aee61b742f96f9f1503ebce9f947d6f1a not found: ID does not exist"
Jul 27 17:31:02.056665 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:02.056639    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb689474d-6hs6g"]
Jul 27 17:31:02.058676 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:02.058653    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cb689474d-6hs6g"]
Jul 27 17:31:03.119498 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:03.119463    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef5cdc13-4132-4804-bd5e-a5d068897bdf" path="/var/lib/kubelet/pods/ef5cdc13-4132-4804-bd5e-a5d068897bdf/volumes"
Jul 27 17:31:05.586459 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:05.586420    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:07.246664 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:07.246628    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666b8c6bcb-f2nbx"]
Jul 27 17:31:07.473528 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:07.473493    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:17.510663 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:17.510628    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:18.033611 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.033580    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:31:18.033883 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.033843    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="prometheus" containerID="cri-o://ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f" gracePeriod=600
Jul 27 17:31:18.034026 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.033880    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy" containerID="cri-o://b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987" gracePeriod=600
Jul 27 17:31:18.034026 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.033905    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy-web" containerID="cri-o://260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d" gracePeriod=600
Jul 27 17:31:18.034548 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.034490    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="thanos-sidecar" containerID="cri-o://96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f" gracePeriod=600
Jul 27 17:31:18.034683 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.034496    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f" gracePeriod=600
Jul 27 17:31:18.035210 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.035124    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="config-reloader" containerID="cri-o://2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b" gracePeriod=600
Jul 27 17:31:18.282918 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.282889    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:18.372495 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.372411    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-config\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.372495 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.372448    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-kube-rbac-proxy\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.372495 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.372486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-db\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.372760 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.372509    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-0\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.372760 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.372528    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnk8j\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-kube-api-access-cnk8j\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.372760 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.372552    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-web-config\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.372760 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.372576    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-kubelet-serving-ca-bundle\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.372760 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.372615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-2\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373018 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.372991    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:18.373159 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373140    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373142    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:18.373252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373175    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-grpc-tls\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373354 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373275    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-metrics-client-ca\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373354 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373319    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373457 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-trusted-ca-bundle\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373457 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373393    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-1\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373457 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373429    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-tls\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373611 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373459    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-metrics-client-certs\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373611 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373486    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-thanos-prometheus-http-client-file\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373611 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373518    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-tls-assets\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373611 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373564    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-serving-certs-ca-bundle\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.373611 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.373590    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-config-out\") pod \"920a00fa-eb6f-46e5-a08a-e073e31a471f\" (UID: \"920a00fa-eb6f-46e5-a08a-e073e31a471f\") "
Jul 27 17:31:18.374663 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.374053    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:18.374663 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.374486    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.374663 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.374512    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.374663 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.374528    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.374663 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.374603    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:18.375016 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.374697    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:18.375016 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.374817    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:18.375309 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.375279    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:18.375437 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.375407    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:18.375508 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.375444    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:18.376256 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.376228    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-kube-api-access-cnk8j" (OuterVolumeSpecName: "kube-api-access-cnk8j") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "kube-api-access-cnk8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:18.376355 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.376335    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-config" (OuterVolumeSpecName: "config") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:18.376775 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.376730    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:18.376864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.376834    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:18.377771 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.377743    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:18.378017 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.377959    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:18.378115 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.378046    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:18.378157 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.378137    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-config-out" (OuterVolumeSpecName: "config-out") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:18.378267 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.378246    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:18.378361 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.378347    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:18.386281 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.386260    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-web-config" (OuterVolumeSpecName: "web-config") pod "920a00fa-eb6f-46e5-a08a-e073e31a471f" (UID: "920a00fa-eb6f-46e5-a08a-e073e31a471f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:18.475017 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.474982    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475017 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475009    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475017 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475020    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475028    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-metrics-client-certs\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475038    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475047    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-tls-assets\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475055    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475064    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-config-out\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475072    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475080    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-kube-rbac-proxy\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475087    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/920a00fa-eb6f-46e5-a08a-e073e31a471f-prometheus-k8s-db\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475095    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnk8j\" (UniqueName: \"kubernetes.io/projected/920a00fa-eb6f-46e5-a08a-e073e31a471f-kube-api-access-cnk8j\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475103    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-web-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475112    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475120    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-grpc-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475128    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/920a00fa-eb6f-46e5-a08a-e073e31a471f-configmap-metrics-client-ca\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:18.475248 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:18.475137    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/920a00fa-eb6f-46e5-a08a-e073e31a471f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:19.102232 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102175    2564 generic.go:356] "Generic (PLEG): container finished" podID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerID="60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f" exitCode=0
Jul 27 17:31:19.102232 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerDied","Data":"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"}
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102260    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102222    2564 generic.go:356] "Generic (PLEG): container finished" podID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerID="b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987" exitCode=0
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerDied","Data":"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"}
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102325    2564 scope.go:117] "RemoveContainer" containerID="60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102338    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerDied","Data":"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"}
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102311    2564 generic.go:356] "Generic (PLEG): container finished" podID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerID="260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d" exitCode=0
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102373    2564 generic.go:356] "Generic (PLEG): container finished" podID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerID="96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f" exitCode=0
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102383    2564 generic.go:356] "Generic (PLEG): container finished" podID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerID="2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b" exitCode=0
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102389    2564 generic.go:356] "Generic (PLEG): container finished" podID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerID="ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f" exitCode=0
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerDied","Data":"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"}
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102503    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerDied","Data":"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"}
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerDied","Data":"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"}
Jul 27 17:31:19.102660 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.102567    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"920a00fa-eb6f-46e5-a08a-e073e31a471f","Type":"ContainerDied","Data":"6a5eb4eab08fce73ad6983410db502e4bc2835298e8014ba878c8992dcf1a0d5"}
Jul 27 17:31:19.110486 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.110463    2564 scope.go:117] "RemoveContainer" containerID="b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"
Jul 27 17:31:19.118057 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.118042    2564 scope.go:117] "RemoveContainer" containerID="260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"
Jul 27 17:31:19.124595 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.124580    2564 scope.go:117] "RemoveContainer" containerID="96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"
Jul 27 17:31:19.129625 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.129593    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:31:19.132032 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.132011    2564 scope.go:117] "RemoveContainer" containerID="2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"
Jul 27 17:31:19.136602 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.136580    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:31:19.139698 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.139682    2564 scope.go:117] "RemoveContainer" containerID="ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"
Jul 27 17:31:19.146753 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.146733    2564 scope.go:117] "RemoveContainer" containerID="979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"
Jul 27 17:31:19.152975 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.152960    2564 scope.go:117] "RemoveContainer" containerID="60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"
Jul 27 17:31:19.153240 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:19.153218    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": container with ID starting with 60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f not found: ID does not exist" containerID="60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"
Jul 27 17:31:19.153310 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.153247    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"} err="failed to get container status \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": rpc error: code = NotFound desc = could not find container \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": container with ID starting with 60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f not found: ID does not exist"
Jul 27 17:31:19.153310 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.153264    2564 scope.go:117] "RemoveContainer" containerID="b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"
Jul 27 17:31:19.153481 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:19.153465    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": container with ID starting with b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987 not found: ID does not exist" containerID="b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"
Jul 27 17:31:19.153525 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.153485    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"} err="failed to get container status \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": rpc error: code = NotFound desc = could not find container \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": container with ID starting with b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987 not found: ID does not exist"
Jul 27 17:31:19.153525 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.153497    2564 scope.go:117] "RemoveContainer" containerID="260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"
Jul 27 17:31:19.153744 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:19.153727    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": container with ID starting with 260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d not found: ID does not exist" containerID="260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"
Jul 27 17:31:19.153804 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.153753    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"} err="failed to get container status \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": rpc error: code = NotFound desc = could not find container \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": container with ID starting with 260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d not found: ID does not exist"
Jul 27 17:31:19.153804 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.153776    2564 scope.go:117] "RemoveContainer" containerID="96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"
Jul 27 17:31:19.153981 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:19.153966    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": container with ID starting with 96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f not found: ID does not exist" containerID="96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"
Jul 27 17:31:19.154023 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.153985    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"} err="failed to get container status \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": rpc error: code = NotFound desc = could not find container \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": container with ID starting with 96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f not found: ID does not exist"
Jul 27 17:31:19.154023 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.153997    2564 scope.go:117] "RemoveContainer" containerID="2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"
Jul 27 17:31:19.154237 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:19.154175    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": container with ID starting with 2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b not found: ID does not exist" containerID="2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"
Jul 27 17:31:19.154286 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.154247    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"} err="failed to get container status \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": rpc error: code = NotFound desc = could not find container \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": container with ID starting with 2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b not found: ID does not exist"
Jul 27 17:31:19.154286 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.154269    2564 scope.go:117] "RemoveContainer" containerID="ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"
Jul 27 17:31:19.154518 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:19.154504    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": container with ID starting with ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f not found: ID does not exist" containerID="ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"
Jul 27 17:31:19.154564 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.154522    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"} err="failed to get container status \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": rpc error: code = NotFound desc = could not find container \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": container with ID starting with ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f not found: ID does not exist"
Jul 27 17:31:19.154564 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.154534    2564 scope.go:117] "RemoveContainer" containerID="979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"
Jul 27 17:31:19.154741 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:19.154723    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": container with ID starting with 979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49 not found: ID does not exist" containerID="979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"
Jul 27 17:31:19.154801 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.154748    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"} err="failed to get container status \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": rpc error: code = NotFound desc = could not find container \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": container with ID starting with 979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49 not found: ID does not exist"
Jul 27 17:31:19.154801 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.154767    2564 scope.go:117] "RemoveContainer" containerID="60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"
Jul 27 17:31:19.154989 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.154972    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"} err="failed to get container status \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": rpc error: code = NotFound desc = could not find container \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": container with ID starting with 60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f not found: ID does not exist"
Jul 27 17:31:19.155055 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.154990    2564 scope.go:117] "RemoveContainer" containerID="b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"
Jul 27 17:31:19.155243 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.155223    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"} err="failed to get container status \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": rpc error: code = NotFound desc = could not find container \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": container with ID starting with b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987 not found: ID does not exist"
Jul 27 17:31:19.155293 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.155244    2564 scope.go:117] "RemoveContainer" containerID="260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"
Jul 27 17:31:19.155442 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.155424    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"} err="failed to get container status \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": rpc error: code = NotFound desc = could not find container \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": container with ID starting with 260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d not found: ID does not exist"
Jul 27 17:31:19.155486 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.155444    2564 scope.go:117] "RemoveContainer" containerID="96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"
Jul 27 17:31:19.155659 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.155642    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"} err="failed to get container status \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": rpc error: code = NotFound desc = could not find container \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": container with ID starting with 96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f not found: ID does not exist"
Jul 27 17:31:19.155724 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.155660    2564 scope.go:117] "RemoveContainer" containerID="2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"
Jul 27 17:31:19.155864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.155844    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"} err="failed to get container status \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": rpc error: code = NotFound desc = could not find container \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": container with ID starting with 2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b not found: ID does not exist"
Jul 27 17:31:19.155909 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.155866    2564 scope.go:117] "RemoveContainer" containerID="ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"
Jul 27 17:31:19.156052 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156034    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"} err="failed to get container status \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": rpc error: code = NotFound desc = could not find container \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": container with ID starting with ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f not found: ID does not exist"
Jul 27 17:31:19.156121 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156052    2564 scope.go:117] "RemoveContainer" containerID="979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"
Jul 27 17:31:19.156305 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156284    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"} err="failed to get container status \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": rpc error: code = NotFound desc = could not find container \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": container with ID starting with 979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49 not found: ID does not exist"
Jul 27 17:31:19.156355 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156308    2564 scope.go:117] "RemoveContainer" containerID="60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"
Jul 27 17:31:19.156513 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156495    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"} err="failed to get container status \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": rpc error: code = NotFound desc = could not find container \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": container with ID starting with 60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f not found: ID does not exist"
Jul 27 17:31:19.156572 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156515    2564 scope.go:117] "RemoveContainer" containerID="b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"
Jul 27 17:31:19.156728 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156710    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"} err="failed to get container status \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": rpc error: code = NotFound desc = could not find container \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": container with ID starting with b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987 not found: ID does not exist"
Jul 27 17:31:19.156775 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156728    2564 scope.go:117] "RemoveContainer" containerID="260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"
Jul 27 17:31:19.156907 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156888    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"} err="failed to get container status \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": rpc error: code = NotFound desc = could not find container \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": container with ID starting with 260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d not found: ID does not exist"
Jul 27 17:31:19.156972 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.156908    2564 scope.go:117] "RemoveContainer" containerID="96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"
Jul 27 17:31:19.157084 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.157066    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"} err="failed to get container status \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": rpc error: code = NotFound desc = could not find container \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": container with ID starting with 96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f not found: ID does not exist"
Jul 27 17:31:19.157147 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.157086    2564 scope.go:117] "RemoveContainer" containerID="2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"
Jul 27 17:31:19.157353 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.157333    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"} err="failed to get container status \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": rpc error: code = NotFound desc = could not find container \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": container with ID starting with 2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b not found: ID does not exist"
Jul 27 17:31:19.157397 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.157354    2564 scope.go:117] "RemoveContainer" containerID="ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"
Jul 27 17:31:19.157567 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.157549    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"} err="failed to get container status \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": rpc error: code = NotFound desc = could not find container \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": container with ID starting with ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f not found: ID does not exist"
Jul 27 17:31:19.157626 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.157568    2564 scope.go:117] "RemoveContainer" containerID="979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"
Jul 27 17:31:19.157804 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.157786    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"} err="failed to get container status \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": rpc error: code = NotFound desc = could not find container \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": container with ID starting with 979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49 not found: ID does not exist"
Jul 27 17:31:19.157846 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.157805    2564 scope.go:117] "RemoveContainer" containerID="60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"
Jul 27 17:31:19.158009 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.157992    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"} err="failed to get container status \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": rpc error: code = NotFound desc = could not find container \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": container with ID starting with 60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f not found: ID does not exist"
Jul 27 17:31:19.158067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.158010    2564 scope.go:117] "RemoveContainer" containerID="b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"
Jul 27 17:31:19.158243 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.158223    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"} err="failed to get container status \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": rpc error: code = NotFound desc = could not find container \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": container with ID starting with b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987 not found: ID does not exist"
Jul 27 17:31:19.158289 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.158245    2564 scope.go:117] "RemoveContainer" containerID="260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"
Jul 27 17:31:19.158421 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.158397    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"} err="failed to get container status \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": rpc error: code = NotFound desc = could not find container \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": container with ID starting with 260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d not found: ID does not exist"
Jul 27 17:31:19.158421 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.158414    2564 scope.go:117] "RemoveContainer" containerID="96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"
Jul 27 17:31:19.158635 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.158616    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"} err="failed to get container status \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": rpc error: code = NotFound desc = could not find container \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": container with ID starting with 96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f not found: ID does not exist"
Jul 27 17:31:19.158685 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.158636    2564 scope.go:117] "RemoveContainer" containerID="2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"
Jul 27 17:31:19.158829 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.158809    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"} err="failed to get container status \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": rpc error: code = NotFound desc = could not find container \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": container with ID starting with 2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b not found: ID does not exist"
Jul 27 17:31:19.158872 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.158830    2564 scope.go:117] "RemoveContainer" containerID="ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"
Jul 27 17:31:19.159028 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.159010    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"} err="failed to get container status \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": rpc error: code = NotFound desc = could not find container \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": container with ID starting with ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f not found: ID does not exist"
Jul 27 17:31:19.159074 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.159029    2564 scope.go:117] "RemoveContainer" containerID="979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"
Jul 27 17:31:19.159296 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.159274    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"} err="failed to get container status \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": rpc error: code = NotFound desc = could not find container \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": container with ID starting with 979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49 not found: ID does not exist"
Jul 27 17:31:19.159296 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.159294    2564 scope.go:117] "RemoveContainer" containerID="60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"
Jul 27 17:31:19.159551 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.159532    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"} err="failed to get container status \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": rpc error: code = NotFound desc = could not find container \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": container with ID starting with 60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f not found: ID does not exist"
Jul 27 17:31:19.159618 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.159553    2564 scope.go:117] "RemoveContainer" containerID="b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"
Jul 27 17:31:19.159800 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.159776    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"} err="failed to get container status \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": rpc error: code = NotFound desc = could not find container \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": container with ID starting with b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987 not found: ID does not exist"
Jul 27 17:31:19.159800 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.159797    2564 scope.go:117] "RemoveContainer" containerID="260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"
Jul 27 17:31:19.160043 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.160025    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"} err="failed to get container status \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": rpc error: code = NotFound desc = could not find container \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": container with ID starting with 260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d not found: ID does not exist"
Jul 27 17:31:19.160043 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.160044    2564 scope.go:117] "RemoveContainer" containerID="96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"
Jul 27 17:31:19.160399 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.160379    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"} err="failed to get container status \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": rpc error: code = NotFound desc = could not find container \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": container with ID starting with 96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f not found: ID does not exist"
Jul 27 17:31:19.160399 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.160399    2564 scope.go:117] "RemoveContainer" containerID="2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"
Jul 27 17:31:19.160638 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.160614    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"} err="failed to get container status \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": rpc error: code = NotFound desc = could not find container \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": container with ID starting with 2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b not found: ID does not exist"
Jul 27 17:31:19.160638 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.160638    2564 scope.go:117] "RemoveContainer" containerID="ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"
Jul 27 17:31:19.160889 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.160870    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"} err="failed to get container status \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": rpc error: code = NotFound desc = could not find container \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": container with ID starting with ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f not found: ID does not exist"
Jul 27 17:31:19.160889 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.160889    2564 scope.go:117] "RemoveContainer" containerID="979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"
Jul 27 17:31:19.161104 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.161082    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"} err="failed to get container status \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": rpc error: code = NotFound desc = could not find container \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": container with ID starting with 979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49 not found: ID does not exist"
Jul 27 17:31:19.161231 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.161105    2564 scope.go:117] "RemoveContainer" containerID="60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"
Jul 27 17:31:19.161363 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.161345    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f"} err="failed to get container status \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": rpc error: code = NotFound desc = could not find container \"60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f\": container with ID starting with 60e89d0d601fe6b9f80032de8a4ede3b94e76bb4d595fad83212ea3fc489359f not found: ID does not exist"
Jul 27 17:31:19.161406 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.161363    2564 scope.go:117] "RemoveContainer" containerID="b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"
Jul 27 17:31:19.161571 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.161553    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987"} err="failed to get container status \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": rpc error: code = NotFound desc = could not find container \"b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987\": container with ID starting with b395a99e3ae01cba909176e3f1028c6cb1b5e821c74d018cc3a3c6b93c6fb987 not found: ID does not exist"
Jul 27 17:31:19.161640 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.161574    2564 scope.go:117] "RemoveContainer" containerID="260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"
Jul 27 17:31:19.161838 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.161820    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d"} err="failed to get container status \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": rpc error: code = NotFound desc = could not find container \"260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d\": container with ID starting with 260e6538e83937dccaa0df760c3bcec594cbfcb8e84f4103991e49354a08c02d not found: ID does not exist"
Jul 27 17:31:19.161883 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.161839    2564 scope.go:117] "RemoveContainer" containerID="96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"
Jul 27 17:31:19.162060 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.162040    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f"} err="failed to get container status \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": rpc error: code = NotFound desc = could not find container \"96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f\": container with ID starting with 96905e34848b36a609f8ffc72504469be82961600fdd32e20a8bc6062225483f not found: ID does not exist"
Jul 27 17:31:19.162134 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.162061    2564 scope.go:117] "RemoveContainer" containerID="2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"
Jul 27 17:31:19.162279 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.162259    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b"} err="failed to get container status \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": rpc error: code = NotFound desc = could not find container \"2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b\": container with ID starting with 2df5b0177cd9fc8012fbef449977fd88320e373c283d37bb414a75b3bf5f7b5b not found: ID does not exist"
Jul 27 17:31:19.162279 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.162277    2564 scope.go:117] "RemoveContainer" containerID="ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"
Jul 27 17:31:19.162495 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.162476    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f"} err="failed to get container status \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": rpc error: code = NotFound desc = could not find container \"ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f\": container with ID starting with ad1b11e6b08fa0dd8abe91e170719ffe26d72fe9060ea62c30c3b6edcf816b7f not found: ID does not exist"
Jul 27 17:31:19.162568 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.162496    2564 scope.go:117] "RemoveContainer" containerID="979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"
Jul 27 17:31:19.162725 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.162705    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49"} err="failed to get container status \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": rpc error: code = NotFound desc = could not find container \"979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49\": container with ID starting with 979714efa445e62f35c6e2a6d27db3e188008c405f7ed1fe21c92a73be084e49 not found: ID does not exist"
Jul 27 17:31:19.171760 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.171740    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:31:19.172028 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172012    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="config-reloader"
Jul 27 17:31:19.172102 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172030    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="config-reloader"
Jul 27 17:31:19.172102 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172046    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="prometheus"
Jul 27 17:31:19.172102 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172055    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="prometheus"
Jul 27 17:31:19.172102 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172068    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="init-config-reloader"
Jul 27 17:31:19.172102 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172077    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="init-config-reloader"
Jul 27 17:31:19.172102 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172090    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5cdc13-4132-4804-bd5e-a5d068897bdf" containerName="console"
Jul 27 17:31:19.172102 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172099    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5cdc13-4132-4804-bd5e-a5d068897bdf" containerName="console"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172108    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ecc8a0e-47c6-458f-8862-c787e6d59f1e" containerName="console"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172116    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ecc8a0e-47c6-458f-8862-c787e6d59f1e" containerName="console"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172129    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172138    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172147    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy-thanos"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172155    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy-thanos"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172176    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy-web"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172200    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy-web"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172212    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="thanos-sidecar"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172220    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="thanos-sidecar"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172230    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b4dec9-4516-48fd-ad49-5f4456c8288d" containerName="console"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172238    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b4dec9-4516-48fd-ad49-5f4456c8288d" containerName="console"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172325    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy-web"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172337    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="prometheus"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172346    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy-thanos"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172356    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ecc8a0e-47c6-458f-8862-c787e6d59f1e" containerName="console"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172366    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="config-reloader"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172376    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b4dec9-4516-48fd-ad49-5f4456c8288d" containerName="console"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172385    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5cdc13-4132-4804-bd5e-a5d068897bdf" containerName="console"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172394    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="thanos-sidecar"
Jul 27 17:31:19.172453 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.172403    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" containerName="kube-rbac-proxy"
Jul 27 17:31:19.176139 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.176123    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.180020 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.180002    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 17:31:19.180298 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.180279    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 17:31:19.180364 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.180329    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 17:31:19.180523 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.180507    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 17:31:19.180589 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.180569    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 17:31:19.180589 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.180512    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 17:31:19.181008 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.180988    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cm66b9h8m01hv\""
Jul 27 17:31:19.181091 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.181054    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mwzr8\""
Jul 27 17:31:19.181148 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.181056    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 17:31:19.181148 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.181055    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 17:31:19.181298 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.181123    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 17:31:19.181381 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.181363    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 17:31:19.181448 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.181388    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 17:31:19.182014 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.181977    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 17:31:19.185517 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.185500    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 17:31:19.190492 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.190475    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 17:31:19.195714 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.195694    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:31:19.280994 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.280969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281106 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281106 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281020    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281106 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281036    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281106 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281058    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281106 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4dca4049-bdfd-4125-a942-1ad38c9ea87e-config-out\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281297 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281111    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281297 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bmk6\" (UniqueName: \"kubernetes.io/projected/4dca4049-bdfd-4125-a942-1ad38c9ea87e-kube-api-access-4bmk6\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281297 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4dca4049-bdfd-4125-a942-1ad38c9ea87e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281297 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281297 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281179    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281297 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281247    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281297 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281496 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281496 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-web-config\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281496 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-config\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281496 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281496 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281496 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.281496 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.281485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.382587 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382502    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.382587 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.382766 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.382766 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.382766 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.382766 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.382766 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.382766 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4dca4049-bdfd-4125-a942-1ad38c9ea87e-config-out\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382822    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bmk6\" (UniqueName: \"kubernetes.io/projected/4dca4049-bdfd-4125-a942-1ad38c9ea87e-kube-api-access-4bmk6\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4dca4049-bdfd-4125-a942-1ad38c9ea87e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382884    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.382989    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.383031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383067 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.383067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-web-config\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383529 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.383102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-config\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383529 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.383339    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.383529 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.383425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.385416 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.384439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.385416 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.385156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.386357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.385794    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.386357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.385864    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.386357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.385899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.386357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.385948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.386357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.385959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.386357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.385983    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.386357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.385996    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4dca4049-bdfd-4125-a942-1ad38c9ea87e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.386357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.386011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4dca4049-bdfd-4125-a942-1ad38c9ea87e-config-out\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.386357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.386053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.387290 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.386609    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.387290 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.386650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.387290 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.387242    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.388052 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.388021    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.388140 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.388123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-config\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.388606 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.388585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.388776 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.388760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4dca4049-bdfd-4125-a942-1ad38c9ea87e-web-config\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.389558 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.389540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4dca4049-bdfd-4125-a942-1ad38c9ea87e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.392560 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.392541    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bmk6\" (UniqueName: \"kubernetes.io/projected/4dca4049-bdfd-4125-a942-1ad38c9ea87e-kube-api-access-4bmk6\") pod \"prometheus-k8s-0\" (UID: \"4dca4049-bdfd-4125-a942-1ad38c9ea87e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.485243 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.485178    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:19.626211 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:19.625894    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:31:19.630754 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:31:19.630728    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dca4049_bdfd_4125_a942_1ad38c9ea87e.slice/crio-480f85c1ace4441d6644df353363b150310cf20b724383f43b04ad2fda969d25 WatchSource:0}: Error finding container 480f85c1ace4441d6644df353363b150310cf20b724383f43b04ad2fda969d25: Status 404 returned error can't find the container with id 480f85c1ace4441d6644df353363b150310cf20b724383f43b04ad2fda969d25
Jul 27 17:31:20.109199 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:20.109142    2564 generic.go:356] "Generic (PLEG): container finished" podID="4dca4049-bdfd-4125-a942-1ad38c9ea87e" containerID="283950c070ddb1cc6de83e6ee3e483401ce34bd28f4bd766e93a3cc504f2a611" exitCode=0
Jul 27 17:31:20.109581 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:20.109211    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4dca4049-bdfd-4125-a942-1ad38c9ea87e","Type":"ContainerDied","Data":"283950c070ddb1cc6de83e6ee3e483401ce34bd28f4bd766e93a3cc504f2a611"}
Jul 27 17:31:20.109581 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:20.109271    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4dca4049-bdfd-4125-a942-1ad38c9ea87e","Type":"ContainerStarted","Data":"480f85c1ace4441d6644df353363b150310cf20b724383f43b04ad2fda969d25"}
Jul 27 17:31:20.594443 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:20.594410    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:21.119575 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:21.119545    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920a00fa-eb6f-46e5-a08a-e073e31a471f" path="/var/lib/kubelet/pods/920a00fa-eb6f-46e5-a08a-e073e31a471f/volumes"
Jul 27 17:31:21.120040 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:21.120024    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4dca4049-bdfd-4125-a942-1ad38c9ea87e","Type":"ContainerStarted","Data":"907a7c0609119cd8879ae17434a00065fefaca9e6281261435cbe0d8b0194d65"}
Jul 27 17:31:21.120086 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:21.120075    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4dca4049-bdfd-4125-a942-1ad38c9ea87e","Type":"ContainerStarted","Data":"cc8c8dc8e305ee98ee584013fabca410bd2f3a2bdf3dd4ccf5d885fdf00e4b3c"}
Jul 27 17:31:21.120139 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:21.120107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4dca4049-bdfd-4125-a942-1ad38c9ea87e","Type":"ContainerStarted","Data":"290e90c1ca84b8f65a02499efe49733057afb1597e8728adf8b9dca34847598d"}
Jul 27 17:31:21.120139 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:21.120137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4dca4049-bdfd-4125-a942-1ad38c9ea87e","Type":"ContainerStarted","Data":"4591129db305809f70e5cf140c5a473669d7b2c0943b3b6c15baa78f4a40e01f"}
Jul 27 17:31:21.120213 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:21.120164    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4dca4049-bdfd-4125-a942-1ad38c9ea87e","Type":"ContainerStarted","Data":"2a5976a57e8c0ef4816b5acf3e325251d4b6463cc623555530f3f770eea15391"}
Jul 27 17:31:21.120213 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:21.120207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4dca4049-bdfd-4125-a942-1ad38c9ea87e","Type":"ContainerStarted","Data":"cab6910b380b3c17b69a248eb7539a76851bc9ece44a7f86e6861b92c612672f"}
Jul 27 17:31:21.154040 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:21.153992    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.140440317 podStartE2EDuration="2.153976556s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.536239ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:21.150860814 +0000 UTC m=+224.603481961" watchObservedRunningTime="2026-07-27 17:31:21.153976556 +0000 UTC m=+224.606597702"
Jul 27 17:31:24.485911 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:24.485864    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:31:27.548414 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:27.548376    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:32.265640 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.265580    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx" podUID="9abb877f-c36e-4524-bf95-241c606335ce" containerName="registry" containerID="cri-o://09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d" gracePeriod=30
Jul 27 17:31:32.515812 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.515758    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:31:32.600664 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.600631    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-installation-pull-secrets\") pod \"9abb877f-c36e-4524-bf95-241c606335ce\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") "
Jul 27 17:31:32.600664 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.600665    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abb877f-c36e-4524-bf95-241c606335ce-ca-trust-extracted\") pod \"9abb877f-c36e-4524-bf95-241c606335ce\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") "
Jul 27 17:31:32.600886 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.600721    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") pod \"9abb877f-c36e-4524-bf95-241c606335ce\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") "
Jul 27 17:31:32.600886 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.600744    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-image-registry-private-configuration\") pod \"9abb877f-c36e-4524-bf95-241c606335ce\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") "
Jul 27 17:31:32.600886 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.600786    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-registry-certificates\") pod \"9abb877f-c36e-4524-bf95-241c606335ce\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") "
Jul 27 17:31:32.600886 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.600815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-bound-sa-token\") pod \"9abb877f-c36e-4524-bf95-241c606335ce\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") "
Jul 27 17:31:32.600886 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.600840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c27wn\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-kube-api-access-c27wn\") pod \"9abb877f-c36e-4524-bf95-241c606335ce\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") "
Jul 27 17:31:32.600886 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.600879    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-trusted-ca\") pod \"9abb877f-c36e-4524-bf95-241c606335ce\" (UID: \"9abb877f-c36e-4524-bf95-241c606335ce\") "
Jul 27 17:31:32.601306 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.601252    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9abb877f-c36e-4524-bf95-241c606335ce" (UID: "9abb877f-c36e-4524-bf95-241c606335ce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:32.601486 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.601462    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9abb877f-c36e-4524-bf95-241c606335ce" (UID: "9abb877f-c36e-4524-bf95-241c606335ce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:32.603106 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.603080    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9abb877f-c36e-4524-bf95-241c606335ce" (UID: "9abb877f-c36e-4524-bf95-241c606335ce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:32.603282 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.603257    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-kube-api-access-c27wn" (OuterVolumeSpecName: "kube-api-access-c27wn") pod "9abb877f-c36e-4524-bf95-241c606335ce" (UID: "9abb877f-c36e-4524-bf95-241c606335ce"). InnerVolumeSpecName "kube-api-access-c27wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:32.603392 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.603372    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9abb877f-c36e-4524-bf95-241c606335ce" (UID: "9abb877f-c36e-4524-bf95-241c606335ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:32.603512 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.603493    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9abb877f-c36e-4524-bf95-241c606335ce" (UID: "9abb877f-c36e-4524-bf95-241c606335ce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:32.603718 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.603690    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9abb877f-c36e-4524-bf95-241c606335ce" (UID: "9abb877f-c36e-4524-bf95-241c606335ce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:32.609631 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.609606    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abb877f-c36e-4524-bf95-241c606335ce-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9abb877f-c36e-4524-bf95-241c606335ce" (UID: "9abb877f-c36e-4524-bf95-241c606335ce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:32.702251 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.702203    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-registry-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:32.702251 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.702250    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-image-registry-private-configuration\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:32.702251 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.702263    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-registry-certificates\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:32.702461 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.702273    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-bound-sa-token\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:32.702461 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.702281    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c27wn\" (UniqueName: \"kubernetes.io/projected/9abb877f-c36e-4524-bf95-241c606335ce-kube-api-access-c27wn\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:32.702461 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.702289    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9abb877f-c36e-4524-bf95-241c606335ce-trusted-ca\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:32.702461 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.702297    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9abb877f-c36e-4524-bf95-241c606335ce-installation-pull-secrets\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:32.702461 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:32.702305    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9abb877f-c36e-4524-bf95-241c606335ce-ca-trust-extracted\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:31:33.162232 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:33.162201    2564 generic.go:356] "Generic (PLEG): container finished" podID="9abb877f-c36e-4524-bf95-241c606335ce" containerID="09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d" exitCode=0
Jul 27 17:31:33.162376 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:33.162248    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx"
Jul 27 17:31:33.162376 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:33.162272    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx" event={"ID":"9abb877f-c36e-4524-bf95-241c606335ce","Type":"ContainerDied","Data":"09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d"}
Jul 27 17:31:33.162376 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:33.162315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666b8c6bcb-f2nbx" event={"ID":"9abb877f-c36e-4524-bf95-241c606335ce","Type":"ContainerDied","Data":"a39612e61f29c9573e56a1fdc5318609dcb722269b9a79dc52eb1b7adee66690"}
Jul 27 17:31:33.162376 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:33.162339    2564 scope.go:117] "RemoveContainer" containerID="09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d"
Jul 27 17:31:33.170340 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:33.170321    2564 scope.go:117] "RemoveContainer" containerID="09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d"
Jul 27 17:31:33.170618 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:33.170589    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d\": container with ID starting with 09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d not found: ID does not exist" containerID="09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d"
Jul 27 17:31:33.170683 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:33.170626    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d"} err="failed to get container status \"09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d\": rpc error: code = NotFound desc = could not find container \"09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d\": container with ID starting with 09f3c09c2d1df290caba1f25d11a9c097d0c209e59c0dd88baecf36cc89b452d not found: ID does not exist"
Jul 27 17:31:33.181654 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:33.181632    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666b8c6bcb-f2nbx"]
Jul 27 17:31:33.185062 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:33.185040    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-666b8c6bcb-f2nbx"]
Jul 27 17:31:35.119137 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:31:35.119107    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9abb877f-c36e-4524-bf95-241c606335ce" path="/var/lib/kubelet/pods/9abb877f-c36e-4524-bf95-241c606335ce/volumes"
Jul 27 17:31:35.586586 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:35.586553    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:35.586759 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:35.586653    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:35.595960 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:31:35.595928    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4dec9_4516_48fd_ad49_5f4456c8288d.slice/crio-15a9dbb6c8fe24d6ef4896592dcd6c57762392b202edcda6787de058f10bc0a4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc8a0e_47c6_458f_8862_c787e6d59f1e.slice\": RecentStats: unable to find data in memory cache]"
Jul 27 17:32:19.485971 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:32:19.485927    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:32:19.502762 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:32:19.502735    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:32:20.354994 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:32:20.354967    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:32:36.920139 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:32:36.920113    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:32:36.922409 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:32:36.922387    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:32:36.929476 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:32:36.929453    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:32:36.930761 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:32:36.930742    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:32:36.932129 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:32:36.932113    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 17:34:06.685012 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.684979    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-d8dz4"]
Jul 27 17:34:06.685506 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.685305    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abb877f-c36e-4524-bf95-241c606335ce" containerName="registry"
Jul 27 17:34:06.685506 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.685318    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abb877f-c36e-4524-bf95-241c606335ce" containerName="registry"
Jul 27 17:34:06.685506 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.685362    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abb877f-c36e-4524-bf95-241c606335ce" containerName="registry"
Jul 27 17:34:06.687395 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.687374    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:34:06.690565 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.690540    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 17:34:06.690565 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.690551    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 27 17:34:06.690723 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.690542    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wrc7l\""
Jul 27 17:34:06.690723 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.690570    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 17:34:06.697314 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.697294    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d8dz4"]
Jul 27 17:34:06.798041 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.798014    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/32320e1d-7049-4caa-ab3a-f75ee8a8dd1c-data\") pod \"seaweedfs-64568bcd49-d8dz4\" (UID: \"32320e1d-7049-4caa-ab3a-f75ee8a8dd1c\") " pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:34:06.798246 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.798057    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtqqw\" (UniqueName: \"kubernetes.io/projected/32320e1d-7049-4caa-ab3a-f75ee8a8dd1c-kube-api-access-wtqqw\") pod \"seaweedfs-64568bcd49-d8dz4\" (UID: \"32320e1d-7049-4caa-ab3a-f75ee8a8dd1c\") " pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:34:06.899252 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.899218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtqqw\" (UniqueName: \"kubernetes.io/projected/32320e1d-7049-4caa-ab3a-f75ee8a8dd1c-kube-api-access-wtqqw\") pod \"seaweedfs-64568bcd49-d8dz4\" (UID: \"32320e1d-7049-4caa-ab3a-f75ee8a8dd1c\") " pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:34:06.899410 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.899318    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/32320e1d-7049-4caa-ab3a-f75ee8a8dd1c-data\") pod \"seaweedfs-64568bcd49-d8dz4\" (UID: \"32320e1d-7049-4caa-ab3a-f75ee8a8dd1c\") " pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:34:06.899655 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.899636    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/32320e1d-7049-4caa-ab3a-f75ee8a8dd1c-data\") pod \"seaweedfs-64568bcd49-d8dz4\" (UID: \"32320e1d-7049-4caa-ab3a-f75ee8a8dd1c\") " pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:34:06.907050 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.907023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtqqw\" (UniqueName: \"kubernetes.io/projected/32320e1d-7049-4caa-ab3a-f75ee8a8dd1c-kube-api-access-wtqqw\") pod \"seaweedfs-64568bcd49-d8dz4\" (UID: \"32320e1d-7049-4caa-ab3a-f75ee8a8dd1c\") " pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:34:06.997883 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:06.997852    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:34:07.122822 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:07.122671    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d8dz4"]
Jul 27 17:34:07.125219 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:34:07.125193    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32320e1d_7049_4caa_ab3a_f75ee8a8dd1c.slice/crio-be846b16a15fc87a4fa24ef20eb7acd7f2825a15ea052a885aae3cc87b126364 WatchSource:0}: Error finding container be846b16a15fc87a4fa24ef20eb7acd7f2825a15ea052a885aae3cc87b126364: Status 404 returned error can't find the container with id be846b16a15fc87a4fa24ef20eb7acd7f2825a15ea052a885aae3cc87b126364
Jul 27 17:34:07.126333 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:07.126317    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:34:07.739132 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:07.739101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d8dz4" event={"ID":"32320e1d-7049-4caa-ab3a-f75ee8a8dd1c","Type":"ContainerStarted","Data":"be846b16a15fc87a4fa24ef20eb7acd7f2825a15ea052a885aae3cc87b126364"}
Jul 27 17:34:10.752988 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:10.752936    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d8dz4" event={"ID":"32320e1d-7049-4caa-ab3a-f75ee8a8dd1c","Type":"ContainerStarted","Data":"fb8fbe82d3619f3af4fbeca236d8c0406cb32ba41caffa4e89a68f528255b33b"}
Jul 27 17:34:10.753349 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:10.753081    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:34:10.771140 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:10.771089    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-d8dz4" podStartSLOduration=1.2486634460000001 podStartE2EDuration="4.771074097s" totalImagesPullingTime="3.522410651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:34:10.770080181 +0000 UTC m=+394.222701326" watchObservedRunningTime="2026-07-27 17:34:10.771074097 +0000 UTC m=+394.223695243"
Jul 27 17:34:16.758385 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:34:16.758356    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-d8dz4"
Jul 27 17:35:08.016171 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.016091    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f5c7d6b87-6w45f"]
Jul 27 17:35:08.018473 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.018447    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.021564 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.021545    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 17:35:08.021677 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.021590    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d8nfx\""
Jul 27 17:35:08.022092 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.022071    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 17:35:08.022721 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.022699    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 17:35:08.022821 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.022709    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 17:35:08.023584 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.023561    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 17:35:08.030898 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.030876    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 17:35:08.033538 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.033517    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5c7d6b87-6w45f"]
Jul 27 17:35:08.076980 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.076944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpb5\" (UniqueName: \"kubernetes.io/projected/50e0de24-faf3-4ff5-85e5-7052840da433-kube-api-access-wnpb5\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.076980 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.076986    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50e0de24-faf3-4ff5-85e5-7052840da433-console-oauth-config\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.077239 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.077009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50e0de24-faf3-4ff5-85e5-7052840da433-console-serving-cert\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.077239 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.077025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-console-config\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.077239 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.077137    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-service-ca\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.077239 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.077209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-oauth-serving-cert\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.077423 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.077281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-trusted-ca-bundle\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.178128 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.178087    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpb5\" (UniqueName: \"kubernetes.io/projected/50e0de24-faf3-4ff5-85e5-7052840da433-kube-api-access-wnpb5\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.178128 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.178134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50e0de24-faf3-4ff5-85e5-7052840da433-console-oauth-config\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.178414 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.178160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50e0de24-faf3-4ff5-85e5-7052840da433-console-serving-cert\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.178414 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.178178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-console-config\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.178414 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.178230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-service-ca\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.178414 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.178247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-oauth-serving-cert\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.178414 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.178276    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-trusted-ca-bundle\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.179038 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.179000    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-service-ca\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.179154 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.179106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-console-config\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.179242 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.179156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-oauth-serving-cert\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.179306 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.179283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e0de24-faf3-4ff5-85e5-7052840da433-trusted-ca-bundle\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.180817 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.180791    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50e0de24-faf3-4ff5-85e5-7052840da433-console-oauth-config\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.180928 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.180871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50e0de24-faf3-4ff5-85e5-7052840da433-console-serving-cert\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.186851 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.186831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpb5\" (UniqueName: \"kubernetes.io/projected/50e0de24-faf3-4ff5-85e5-7052840da433-kube-api-access-wnpb5\") pod \"console-6f5c7d6b87-6w45f\" (UID: \"50e0de24-faf3-4ff5-85e5-7052840da433\") " pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.331717 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.331622    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:08.461170 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.461133    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5c7d6b87-6w45f"]
Jul 27 17:35:08.464047 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:35:08.464012    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e0de24_faf3_4ff5_85e5_7052840da433.slice/crio-05d784a251874f6e4d02e484fb7bc8de3ff20abf2ed8bb6535c7b5abf0cb1452 WatchSource:0}: Error finding container 05d784a251874f6e4d02e484fb7bc8de3ff20abf2ed8bb6535c7b5abf0cb1452: Status 404 returned error can't find the container with id 05d784a251874f6e4d02e484fb7bc8de3ff20abf2ed8bb6535c7b5abf0cb1452
Jul 27 17:35:08.978501 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.978463    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5c7d6b87-6w45f" event={"ID":"50e0de24-faf3-4ff5-85e5-7052840da433","Type":"ContainerStarted","Data":"eb06883c5a381aeee4e89610dc497ff17eabbb2246c04705cadde2a4c12837eb"}
Jul 27 17:35:08.978691 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.978520    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5c7d6b87-6w45f" event={"ID":"50e0de24-faf3-4ff5-85e5-7052840da433","Type":"ContainerStarted","Data":"05d784a251874f6e4d02e484fb7bc8de3ff20abf2ed8bb6535c7b5abf0cb1452"}
Jul 27 17:35:08.998422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:08.998349    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f5c7d6b87-6w45f" podStartSLOduration=1.998333344 podStartE2EDuration="1.998333344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:08.997373991 +0000 UTC m=+452.449995136" watchObservedRunningTime="2026-07-27 17:35:08.998333344 +0000 UTC m=+452.450954490"
Jul 27 17:35:18.332137 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:18.332100    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:18.332639 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:18.332301    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:18.337077 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:18.337055    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:19.023422 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:19.023395    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5c7d6b87-6w45f"
Jul 27 17:35:38.168420 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.168387    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-bcx6x"]
Jul 27 17:35:38.170669 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.170653    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bcx6x"
Jul 27 17:35:38.179377 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.179355    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bcx6x"]
Jul 27 17:35:38.331537 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.331505    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7kkd\" (UniqueName: \"kubernetes.io/projected/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-kube-api-access-z7kkd\") pod \"s3-init-bcx6x\" (UID: \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\") " pod="kserve/s3-init-bcx6x"
Jul 27 17:35:38.331690 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.331545    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-model-cache\") pod \"s3-init-bcx6x\" (UID: \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\") " pod="kserve/s3-init-bcx6x"
Jul 27 17:35:38.432482 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.432416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7kkd\" (UniqueName: \"kubernetes.io/projected/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-kube-api-access-z7kkd\") pod \"s3-init-bcx6x\" (UID: \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\") " pod="kserve/s3-init-bcx6x"
Jul 27 17:35:38.432482 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.432451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-model-cache\") pod \"s3-init-bcx6x\" (UID: \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\") " pod="kserve/s3-init-bcx6x"
Jul 27 17:35:38.432780 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.432764    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-model-cache\") pod \"s3-init-bcx6x\" (UID: \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\") " pod="kserve/s3-init-bcx6x"
Jul 27 17:35:38.442991 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.442970    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7kkd\" (UniqueName: \"kubernetes.io/projected/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-kube-api-access-z7kkd\") pod \"s3-init-bcx6x\" (UID: \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\") " pod="kserve/s3-init-bcx6x"
Jul 27 17:35:38.479811 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.479790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bcx6x"
Jul 27 17:35:38.605509 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:38.605479    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bcx6x"]
Jul 27 17:35:38.608978 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:35:38.608940    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5fe2eba_eb28_46ea_ba1c_33ce3ab8e78d.slice/crio-dee31771e7559204d54ae927beb6a3c207419d7cf205a9c9655e5a0782d2b297 WatchSource:0}: Error finding container dee31771e7559204d54ae927beb6a3c207419d7cf205a9c9655e5a0782d2b297: Status 404 returned error can't find the container with id dee31771e7559204d54ae927beb6a3c207419d7cf205a9c9655e5a0782d2b297
Jul 27 17:35:39.096243 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:39.096207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bcx6x" event={"ID":"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d","Type":"ContainerStarted","Data":"dee31771e7559204d54ae927beb6a3c207419d7cf205a9c9655e5a0782d2b297"}
Jul 27 17:35:43.122064 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:43.122022    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bcx6x" event={"ID":"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d","Type":"ContainerStarted","Data":"a601d5893fbf03c587dcec9c6a7c7049c1c2d52a861e8f2d69f58995561521d1"}
Jul 27 17:35:47.137482 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:47.137448    2564 generic.go:356] "Generic (PLEG): container finished" podID="d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d" containerID="a601d5893fbf03c587dcec9c6a7c7049c1c2d52a861e8f2d69f58995561521d1" exitCode=0
Jul 27 17:35:47.137864 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:47.137510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bcx6x" event={"ID":"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d","Type":"ContainerDied","Data":"a601d5893fbf03c587dcec9c6a7c7049c1c2d52a861e8f2d69f58995561521d1"}
Jul 27 17:35:52.161322 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:52.161284    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bcx6x" event={"ID":"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d","Type":"ContainerStarted","Data":"44da12feb9a5664a3096f845702f69d4d4cc4031f8be42a5c2e7b61bd7f40ec3"}
Jul 27 17:35:52.181733 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:35:52.181686    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-bcx6x" podStartSLOduration=2.019997493 podStartE2EDuration="14.181670596s" totalImagesPullingTime="8.55311328s" totalInitContainerRuntime="3.608559823s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:52.180939904 +0000 UTC m=+495.633561053" watchObservedRunningTime="2026-07-27 17:35:52.181670596 +0000 UTC m=+495.634291744"
Jul 27 17:36:57.410521 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:57.410482    2564 generic.go:356] "Generic (PLEG): container finished" podID="d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d" containerID="44da12feb9a5664a3096f845702f69d4d4cc4031f8be42a5c2e7b61bd7f40ec3" exitCode=0
Jul 27 17:36:57.410952 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:57.410558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bcx6x" event={"ID":"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d","Type":"ContainerDied","Data":"44da12feb9a5664a3096f845702f69d4d4cc4031f8be42a5c2e7b61bd7f40ec3"}
Jul 27 17:36:58.539624 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:58.539600    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bcx6x"
Jul 27 17:36:58.565098 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:58.565073    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-model-cache\") pod \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\" (UID: \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\") "
Jul 27 17:36:58.584816 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:58.584780    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d" (UID: "d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:58.665844 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:58.665777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7kkd\" (UniqueName: \"kubernetes.io/projected/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-kube-api-access-z7kkd\") pod \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\" (UID: \"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d\") "
Jul 27 17:36:58.665966 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:58.665916    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-model-cache\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:36:58.667793 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:58.667769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-kube-api-access-z7kkd" (OuterVolumeSpecName: "kube-api-access-z7kkd") pod "d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d" (UID: "d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d"). InnerVolumeSpecName "kube-api-access-z7kkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:36:58.766243 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:58.766215    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7kkd\" (UniqueName: \"kubernetes.io/projected/d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d-kube-api-access-z7kkd\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:36:59.418654 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:59.418591    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bcx6x"
Jul 27 17:36:59.418654 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:59.418611    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bcx6x" event={"ID":"d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d","Type":"ContainerDied","Data":"dee31771e7559204d54ae927beb6a3c207419d7cf205a9c9655e5a0782d2b297"}
Jul 27 17:36:59.418654 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:36:59.418646    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee31771e7559204d54ae927beb6a3c207419d7cf205a9c9655e5a0782d2b297"
Jul 27 17:37:36.947905 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:36.947880    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:37:36.948435 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:36.948416    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:37:36.955554 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:36.955533    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:37:36.955800 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:36.955785    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:37:40.147955 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.147917    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-wbpcl"]
Jul 27 17:37:40.148394 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.148219    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d" containerName="download-hf-model"
Jul 27 17:37:40.148394 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.148232    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d" containerName="download-hf-model"
Jul 27 17:37:40.148394 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.148243    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d" containerName="s3-init"
Jul 27 17:37:40.148394 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.148248    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d" containerName="s3-init"
Jul 27 17:37:40.148394 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.148300    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d" containerName="s3-init"
Jul 27 17:37:40.150170 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.150152    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:40.153073 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.153046    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 27 17:37:40.159487 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.159467    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-wbpcl"]
Jul 27 17:37:40.191420 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.191398    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74538621-237b-4ae6-bea4-6f4b6127fb93-model-cache\") pod \"s3-tls-init-custom-wbpcl\" (UID: \"74538621-237b-4ae6-bea4-6f4b6127fb93\") " pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:40.191538 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.191459    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflqs\" (UniqueName: \"kubernetes.io/projected/74538621-237b-4ae6-bea4-6f4b6127fb93-kube-api-access-lflqs\") pod \"s3-tls-init-custom-wbpcl\" (UID: \"74538621-237b-4ae6-bea4-6f4b6127fb93\") " pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:40.292268 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.292247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74538621-237b-4ae6-bea4-6f4b6127fb93-model-cache\") pod \"s3-tls-init-custom-wbpcl\" (UID: \"74538621-237b-4ae6-bea4-6f4b6127fb93\") " pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:40.292365 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.292295    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflqs\" (UniqueName: \"kubernetes.io/projected/74538621-237b-4ae6-bea4-6f4b6127fb93-kube-api-access-lflqs\") pod \"s3-tls-init-custom-wbpcl\" (UID: \"74538621-237b-4ae6-bea4-6f4b6127fb93\") " pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:40.292614 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.292597    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74538621-237b-4ae6-bea4-6f4b6127fb93-model-cache\") pod \"s3-tls-init-custom-wbpcl\" (UID: \"74538621-237b-4ae6-bea4-6f4b6127fb93\") " pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:40.301671 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.301652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflqs\" (UniqueName: \"kubernetes.io/projected/74538621-237b-4ae6-bea4-6f4b6127fb93-kube-api-access-lflqs\") pod \"s3-tls-init-custom-wbpcl\" (UID: \"74538621-237b-4ae6-bea4-6f4b6127fb93\") " pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:40.459790 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.459732    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:40.584751 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:40.584726    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-wbpcl"]
Jul 27 17:37:40.587012 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:37:40.586991    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74538621_237b_4ae6_bea4_6f4b6127fb93.slice/crio-a64e153aaeef72a771926b9718a53e7bb7321aedc0d4fdd27e6bd91c1a465dc5 WatchSource:0}: Error finding container a64e153aaeef72a771926b9718a53e7bb7321aedc0d4fdd27e6bd91c1a465dc5: Status 404 returned error can't find the container with id a64e153aaeef72a771926b9718a53e7bb7321aedc0d4fdd27e6bd91c1a465dc5
Jul 27 17:37:41.583608 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:41.583572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wbpcl" event={"ID":"74538621-237b-4ae6-bea4-6f4b6127fb93","Type":"ContainerStarted","Data":"d7ec21c830a1e9f004bd7c039551a63f5f636d96bd4f3b8c5c51df5fce4e0ffa"}
Jul 27 17:37:41.583608 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:41.583613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wbpcl" event={"ID":"74538621-237b-4ae6-bea4-6f4b6127fb93","Type":"ContainerStarted","Data":"a64e153aaeef72a771926b9718a53e7bb7321aedc0d4fdd27e6bd91c1a465dc5"}
Jul 27 17:37:43.593043 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:43.593014    2564 generic.go:356] "Generic (PLEG): container finished" podID="74538621-237b-4ae6-bea4-6f4b6127fb93" containerID="d7ec21c830a1e9f004bd7c039551a63f5f636d96bd4f3b8c5c51df5fce4e0ffa" exitCode=0
Jul 27 17:37:43.593423 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:43.593096    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wbpcl" event={"ID":"74538621-237b-4ae6-bea4-6f4b6127fb93","Type":"ContainerDied","Data":"d7ec21c830a1e9f004bd7c039551a63f5f636d96bd4f3b8c5c51df5fce4e0ffa"}
Jul 27 17:37:44.599021 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:44.598983    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wbpcl" event={"ID":"74538621-237b-4ae6-bea4-6f4b6127fb93","Type":"ContainerStarted","Data":"ee21328f730241e9736884245356b0936b55a2e7e972d6d4f22fd9d94e40f423"}
Jul 27 17:37:44.618766 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:44.618715    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-wbpcl" podStartSLOduration=2.203801931 podStartE2EDuration="4.618702196s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.414900265s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:44.617152926 +0000 UTC m=+608.069774073" watchObservedRunningTime="2026-07-27 17:37:44.618702196 +0000 UTC m=+608.071323341"
Jul 27 17:37:50.624504 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:50.624472    2564 generic.go:356] "Generic (PLEG): container finished" podID="74538621-237b-4ae6-bea4-6f4b6127fb93" containerID="ee21328f730241e9736884245356b0936b55a2e7e972d6d4f22fd9d94e40f423" exitCode=0
Jul 27 17:37:50.624906 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:50.624546    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wbpcl" event={"ID":"74538621-237b-4ae6-bea4-6f4b6127fb93","Type":"ContainerDied","Data":"ee21328f730241e9736884245356b0936b55a2e7e972d6d4f22fd9d94e40f423"}
Jul 27 17:37:51.778935 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:51.778910    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:51.871572 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:51.871547    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lflqs\" (UniqueName: \"kubernetes.io/projected/74538621-237b-4ae6-bea4-6f4b6127fb93-kube-api-access-lflqs\") pod \"74538621-237b-4ae6-bea4-6f4b6127fb93\" (UID: \"74538621-237b-4ae6-bea4-6f4b6127fb93\") "
Jul 27 17:37:51.871686 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:51.871615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74538621-237b-4ae6-bea4-6f4b6127fb93-model-cache\") pod \"74538621-237b-4ae6-bea4-6f4b6127fb93\" (UID: \"74538621-237b-4ae6-bea4-6f4b6127fb93\") "
Jul 27 17:37:51.873527 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:51.873505    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74538621-237b-4ae6-bea4-6f4b6127fb93-kube-api-access-lflqs" (OuterVolumeSpecName: "kube-api-access-lflqs") pod "74538621-237b-4ae6-bea4-6f4b6127fb93" (UID: "74538621-237b-4ae6-bea4-6f4b6127fb93"). InnerVolumeSpecName "kube-api-access-lflqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:37:51.891193 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:51.891133    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74538621-237b-4ae6-bea4-6f4b6127fb93-model-cache" (OuterVolumeSpecName: "model-cache") pod "74538621-237b-4ae6-bea4-6f4b6127fb93" (UID: "74538621-237b-4ae6-bea4-6f4b6127fb93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:37:51.972119 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:51.972095    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74538621-237b-4ae6-bea4-6f4b6127fb93-model-cache\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:37:51.972119 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:51.972118    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lflqs\" (UniqueName: \"kubernetes.io/projected/74538621-237b-4ae6-bea4-6f4b6127fb93-kube-api-access-lflqs\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:37:52.633904 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:52.633872    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-wbpcl"
Jul 27 17:37:52.633904 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:52.633884    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wbpcl" event={"ID":"74538621-237b-4ae6-bea4-6f4b6127fb93","Type":"ContainerDied","Data":"a64e153aaeef72a771926b9718a53e7bb7321aedc0d4fdd27e6bd91c1a465dc5"}
Jul 27 17:37:52.634116 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:52.633923    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a64e153aaeef72a771926b9718a53e7bb7321aedc0d4fdd27e6bd91c1a465dc5"
Jul 27 17:37:53.227552 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.227493    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"]
Jul 27 17:37:53.228004 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.227952    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74538621-237b-4ae6-bea4-6f4b6127fb93" containerName="s3-tls-init-custom"
Jul 27 17:37:53.228004 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.227971    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="74538621-237b-4ae6-bea4-6f4b6127fb93" containerName="s3-tls-init-custom"
Jul 27 17:37:53.228004 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.227985    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74538621-237b-4ae6-bea4-6f4b6127fb93" containerName="download-hf-model"
Jul 27 17:37:53.228004 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.227993    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="74538621-237b-4ae6-bea4-6f4b6127fb93" containerName="download-hf-model"
Jul 27 17:37:53.228239 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.228075    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="74538621-237b-4ae6-bea4-6f4b6127fb93" containerName="s3-tls-init-custom"
Jul 27 17:37:53.230070 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.230050    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.232965 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.232945    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 27 17:37:53.233074 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.232948    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 27 17:37:53.239469 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.239447    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"]
Jul 27 17:37:53.279358 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.279336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2tpz\" (UniqueName: \"kubernetes.io/projected/d89c7371-f884-427b-ace9-a7a4dc4424ab-kube-api-access-w2tpz\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.279481 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.279400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d89c7371-f884-427b-ace9-a7a4dc4424ab-data\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.279481 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.279449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/d89c7371-f884-427b-ace9-a7a4dc4424ab-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.379911 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.379878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d89c7371-f884-427b-ace9-a7a4dc4424ab-data\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.380079 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.379919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/d89c7371-f884-427b-ace9-a7a4dc4424ab-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.380079 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.379952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2tpz\" (UniqueName: \"kubernetes.io/projected/d89c7371-f884-427b-ace9-a7a4dc4424ab-kube-api-access-w2tpz\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.380204 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:37:53.380089    2564 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 27 17:37:53.380204 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:37:53.380107    2564 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-49rlc: secret "seaweedfs-tls-serving" not found
Jul 27 17:37:53.380204 ip-10-0-129-58 kubenswrapper[2564]: E0727 17:37:53.380176    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89c7371-f884-427b-ace9-a7a4dc4424ab-seaweedfs-tls-serving podName:d89c7371-f884-427b-ace9-a7a4dc4424ab nodeName:}" failed. No retries permitted until 2026-07-27 17:37:53.880155502 +0000 UTC m=+617.332776633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/d89c7371-f884-427b-ace9-a7a4dc4424ab-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-49rlc" (UID: "d89c7371-f884-427b-ace9-a7a4dc4424ab") : secret "seaweedfs-tls-serving" not found
Jul 27 17:37:53.380362 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.380331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d89c7371-f884-427b-ace9-a7a4dc4424ab-data\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.389214 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.389196    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2tpz\" (UniqueName: \"kubernetes.io/projected/d89c7371-f884-427b-ace9-a7a4dc4424ab-kube-api-access-w2tpz\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.883892 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.883858    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/d89c7371-f884-427b-ace9-a7a4dc4424ab-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:53.886271 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:53.886246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/d89c7371-f884-427b-ace9-a7a4dc4424ab-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-49rlc\" (UID: \"d89c7371-f884-427b-ace9-a7a4dc4424ab\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:54.141228 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:54.141124    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"
Jul 27 17:37:54.268079 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:54.268046    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-49rlc"]
Jul 27 17:37:54.272058 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:37:54.272033    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89c7371_f884_427b_ace9_a7a4dc4424ab.slice/crio-ba143a2dfc1f9bfb8fe8a66932ff9d56134d59a9caeb39c4152dc71ee162d146 WatchSource:0}: Error finding container ba143a2dfc1f9bfb8fe8a66932ff9d56134d59a9caeb39c4152dc71ee162d146: Status 404 returned error can't find the container with id ba143a2dfc1f9bfb8fe8a66932ff9d56134d59a9caeb39c4152dc71ee162d146
Jul 27 17:37:54.644311 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:54.644279    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc" event={"ID":"d89c7371-f884-427b-ace9-a7a4dc4424ab","Type":"ContainerStarted","Data":"199d6f208593efc14ef7e0b968a303a9374e7c7a4bbbe4c3de95b7dfca2177d0"}
Jul 27 17:37:54.644426 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:54.644326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc" event={"ID":"d89c7371-f884-427b-ace9-a7a4dc4424ab","Type":"ContainerStarted","Data":"ba143a2dfc1f9bfb8fe8a66932ff9d56134d59a9caeb39c4152dc71ee162d146"}
Jul 27 17:37:54.661357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:54.661313    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-49rlc" podStartSLOduration=1.4129033 podStartE2EDuration="1.661298908s" totalImagesPullingTime="248.395608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:54.660455651 +0000 UTC m=+618.113076796" watchObservedRunningTime="2026-07-27 17:37:54.661298908 +0000 UTC m=+618.113920054"
Jul 27 17:37:55.205066 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.205036    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-28gp8"]
Jul 27 17:37:55.208113 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.208095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:37:55.216732 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.216708    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-28gp8"]
Jul 27 17:37:55.295735 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.295704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b7a693-8130-461c-a5d8-1363948faa60-model-cache\") pod \"s3-tls-init-serving-28gp8\" (UID: \"f2b7a693-8130-461c-a5d8-1363948faa60\") " pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:37:55.296057 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.295743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clntm\" (UniqueName: \"kubernetes.io/projected/f2b7a693-8130-461c-a5d8-1363948faa60-kube-api-access-clntm\") pod \"s3-tls-init-serving-28gp8\" (UID: \"f2b7a693-8130-461c-a5d8-1363948faa60\") " pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:37:55.396303 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.396280    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b7a693-8130-461c-a5d8-1363948faa60-model-cache\") pod \"s3-tls-init-serving-28gp8\" (UID: \"f2b7a693-8130-461c-a5d8-1363948faa60\") " pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:37:55.396427 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.396316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clntm\" (UniqueName: \"kubernetes.io/projected/f2b7a693-8130-461c-a5d8-1363948faa60-kube-api-access-clntm\") pod \"s3-tls-init-serving-28gp8\" (UID: \"f2b7a693-8130-461c-a5d8-1363948faa60\") " pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:37:55.396623 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.396606    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b7a693-8130-461c-a5d8-1363948faa60-model-cache\") pod \"s3-tls-init-serving-28gp8\" (UID: \"f2b7a693-8130-461c-a5d8-1363948faa60\") " pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:37:55.405194 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.405165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clntm\" (UniqueName: \"kubernetes.io/projected/f2b7a693-8130-461c-a5d8-1363948faa60-kube-api-access-clntm\") pod \"s3-tls-init-serving-28gp8\" (UID: \"f2b7a693-8130-461c-a5d8-1363948faa60\") " pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:37:55.516886 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.516863    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:37:55.644027 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:55.644000    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-28gp8"]
Jul 27 17:37:55.645870 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:37:55.645844    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2b7a693_8130_461c_a5d8_1363948faa60.slice/crio-020aae29e6b5372ad032beb4573dc669d7b64023fa3b87ed37a90df30e1a6400 WatchSource:0}: Error finding container 020aae29e6b5372ad032beb4573dc669d7b64023fa3b87ed37a90df30e1a6400: Status 404 returned error can't find the container with id 020aae29e6b5372ad032beb4573dc669d7b64023fa3b87ed37a90df30e1a6400
Jul 27 17:37:56.655283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:56.655244    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-28gp8" event={"ID":"f2b7a693-8130-461c-a5d8-1363948faa60","Type":"ContainerStarted","Data":"37a54a0d3b1423ca3ecc4b6fbef74a360ccb3a0ffd1c69e5ed362074b8b6a2e7"}
Jul 27 17:37:56.655858 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:56.655294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-28gp8" event={"ID":"f2b7a693-8130-461c-a5d8-1363948faa60","Type":"ContainerStarted","Data":"020aae29e6b5372ad032beb4573dc669d7b64023fa3b87ed37a90df30e1a6400"}
Jul 27 17:37:59.669346 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:59.669313    2564 generic.go:356] "Generic (PLEG): container finished" podID="f2b7a693-8130-461c-a5d8-1363948faa60" containerID="37a54a0d3b1423ca3ecc4b6fbef74a360ccb3a0ffd1c69e5ed362074b8b6a2e7" exitCode=0
Jul 27 17:37:59.669698 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:37:59.669384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-28gp8" event={"ID":"f2b7a693-8130-461c-a5d8-1363948faa60","Type":"ContainerDied","Data":"37a54a0d3b1423ca3ecc4b6fbef74a360ccb3a0ffd1c69e5ed362074b8b6a2e7"}
Jul 27 17:38:00.675060 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:00.675022    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-28gp8" event={"ID":"f2b7a693-8130-461c-a5d8-1363948faa60","Type":"ContainerStarted","Data":"a0ab031db6011342216f634e4514da1945b587ddf8871cb5698c087b7e4f12db"}
Jul 27 17:38:00.692938 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:00.692888    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-28gp8" podStartSLOduration=2.399403074 podStartE2EDuration="5.69287331s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.293470236s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:00.691509775 +0000 UTC m=+624.144130925" watchObservedRunningTime="2026-07-27 17:38:00.69287331 +0000 UTC m=+624.145494468"
Jul 27 17:38:06.704589 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:06.704550    2564 generic.go:356] "Generic (PLEG): container finished" podID="f2b7a693-8130-461c-a5d8-1363948faa60" containerID="a0ab031db6011342216f634e4514da1945b587ddf8871cb5698c087b7e4f12db" exitCode=0
Jul 27 17:38:06.704945 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:06.704621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-28gp8" event={"ID":"f2b7a693-8130-461c-a5d8-1363948faa60","Type":"ContainerDied","Data":"a0ab031db6011342216f634e4514da1945b587ddf8871cb5698c087b7e4f12db"}
Jul 27 17:38:07.844428 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:07.844406    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:38:07.898995 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:07.898955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b7a693-8130-461c-a5d8-1363948faa60-model-cache\") pod \"f2b7a693-8130-461c-a5d8-1363948faa60\" (UID: \"f2b7a693-8130-461c-a5d8-1363948faa60\") "
Jul 27 17:38:07.899141 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:07.899008    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clntm\" (UniqueName: \"kubernetes.io/projected/f2b7a693-8130-461c-a5d8-1363948faa60-kube-api-access-clntm\") pod \"f2b7a693-8130-461c-a5d8-1363948faa60\" (UID: \"f2b7a693-8130-461c-a5d8-1363948faa60\") "
Jul 27 17:38:07.900950 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:07.900921    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b7a693-8130-461c-a5d8-1363948faa60-kube-api-access-clntm" (OuterVolumeSpecName: "kube-api-access-clntm") pod "f2b7a693-8130-461c-a5d8-1363948faa60" (UID: "f2b7a693-8130-461c-a5d8-1363948faa60"). InnerVolumeSpecName "kube-api-access-clntm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:38:07.918094 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:07.918070    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b7a693-8130-461c-a5d8-1363948faa60-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2b7a693-8130-461c-a5d8-1363948faa60" (UID: "f2b7a693-8130-461c-a5d8-1363948faa60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:38:08.000084 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:08.000051    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b7a693-8130-461c-a5d8-1363948faa60-model-cache\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:38:08.000084 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:08.000082    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clntm\" (UniqueName: \"kubernetes.io/projected/f2b7a693-8130-461c-a5d8-1363948faa60-kube-api-access-clntm\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 17:38:08.714141 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:08.714109    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-28gp8"
Jul 27 17:38:08.714141 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:08.714132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-28gp8" event={"ID":"f2b7a693-8130-461c-a5d8-1363948faa60","Type":"ContainerDied","Data":"020aae29e6b5372ad032beb4573dc669d7b64023fa3b87ed37a90df30e1a6400"}
Jul 27 17:38:08.714339 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:38:08.714164    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="020aae29e6b5372ad032beb4573dc669d7b64023fa3b87ed37a90df30e1a6400"
Jul 27 17:42:36.974374 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:42:36.974289    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:42:36.975106 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:42:36.975084    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:42:36.981979 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:42:36.981955    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:42:36.982631 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:42:36.982605    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:47:37.000837 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:47:37.000804    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:47:37.002913 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:47:37.002890    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:47:37.009027 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:47:37.008993    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:47:37.011121 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:47:37.011095    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:52:37.028910 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:52:37.028879    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:52:37.032011 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:52:37.031980    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:52:37.037486 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:52:37.037456    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:52:37.040357 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:52:37.040333    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:57:37.056462 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:57:37.056418    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:57:37.060672 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:57:37.060647    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 17:57:37.064451 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:57:37.064418    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:57:37.068489 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:57:37.068465    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 17:58:06.995161 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:06.995117    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"]
Jul 27 17:58:06.995694 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:06.995537    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b7a693-8130-461c-a5d8-1363948faa60" containerName="s3-tls-init-serving"
Jul 27 17:58:06.995694 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:06.995559    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b7a693-8130-461c-a5d8-1363948faa60" containerName="s3-tls-init-serving"
Jul 27 17:58:06.995694 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:06.995584    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b7a693-8130-461c-a5d8-1363948faa60" containerName="download-hf-model"
Jul 27 17:58:06.995694 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:06.995591    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b7a693-8130-461c-a5d8-1363948faa60" containerName="download-hf-model"
Jul 27 17:58:06.995694 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:06.995661    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b7a693-8130-461c-a5d8-1363948faa60" containerName="s3-tls-init-serving"
Jul 27 17:58:06.998918 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:06.998897    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.002440 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.002357    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 27 17:58:07.002627 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.002550    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 17:58:07.003153 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.003137    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 27 17:58:07.003384 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.003367    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 17:58:07.003579 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.003563    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 17:58:07.003847 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.003830    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 17:58:07.003935 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.003845    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 17:58:07.028393 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.028354    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"]
Jul 27 17:58:07.147590 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.147547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.147590 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.147590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.147825 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.147621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.147825 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.147652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92gtt\" (UniqueName: \"kubernetes.io/projected/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-api-access-92gtt\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.147825 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.147681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1fdec1-4233-48fe-bf05-22fbeb92decf-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.248976 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.248873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.248976 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.248924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.248976 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.248958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.249283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.249013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92gtt\" (UniqueName: \"kubernetes.io/projected/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-api-access-92gtt\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.249283 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.249062    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1fdec1-4233-48fe-bf05-22fbeb92decf-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.249558 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.249535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.249767 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.249747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.249870 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.249853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.251737 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.251712    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1fdec1-4233-48fe-bf05-22fbeb92decf-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.258877 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.258845    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92gtt\" (UniqueName: \"kubernetes.io/projected/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-api-access-92gtt\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gnfs8\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.309892 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.309850    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:07.463594 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.463559    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"]
Jul 27 17:58:07.466343 ip-10-0-129-58 kubenswrapper[2564]: W0727 17:58:07.466310    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1fdec1_4233_48fe_bf05_22fbeb92decf.slice/crio-1a28d84c3802517ff07f77f5c3fa136cfeba184b8f99bd570c36afd506ebd904 WatchSource:0}: Error finding container 1a28d84c3802517ff07f77f5c3fa136cfeba184b8f99bd570c36afd506ebd904: Status 404 returned error can't find the container with id 1a28d84c3802517ff07f77f5c3fa136cfeba184b8f99bd570c36afd506ebd904
Jul 27 17:58:07.714756 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.714709    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" event={"ID":"2c1fdec1-4233-48fe-bf05-22fbeb92decf","Type":"ContainerStarted","Data":"aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6"}
Jul 27 17:58:07.714957 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:07.714792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" event={"ID":"2c1fdec1-4233-48fe-bf05-22fbeb92decf","Type":"ContainerStarted","Data":"1a28d84c3802517ff07f77f5c3fa136cfeba184b8f99bd570c36afd506ebd904"}
Jul 27 17:58:11.731775 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:11.731737    2564 generic.go:356] "Generic (PLEG): container finished" podID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerID="aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6" exitCode=0
Jul 27 17:58:11.731775 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:11.731780    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" event={"ID":"2c1fdec1-4233-48fe-bf05-22fbeb92decf","Type":"ContainerDied","Data":"aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6"}
Jul 27 17:58:11.732991 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:11.732970    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:58:26.822065 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:26.822017    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" event={"ID":"2c1fdec1-4233-48fe-bf05-22fbeb92decf","Type":"ContainerStarted","Data":"dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c"}
Jul 27 17:58:28.833287 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:28.833170    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" event={"ID":"2c1fdec1-4233-48fe-bf05-22fbeb92decf","Type":"ContainerStarted","Data":"5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3"}
Jul 27 17:58:28.833725 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:28.833445    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:28.833725 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:28.833508    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:28.835019 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:28.834984    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:58:28.856170 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:28.856107    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podStartSLOduration=2.315842415 podStartE2EDuration="22.856088675s" totalImagesPullingTime="16.74686043s" totalInitContainerRuntime="3.79338583s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:58:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:58:28.854457442 +0000 UTC m=+1852.307078588" watchObservedRunningTime="2026-07-27 17:58:28.856088675 +0000 UTC m=+1852.308709822"
Jul 27 17:58:29.837841 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:29.837791    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:58:34.842782 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:34.842748    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:58:34.843323 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:34.843286    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:58:44.843865 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:44.843806    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:58:54.844234 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:58:54.844171    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:59:04.843396 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:04.843285    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:59:14.843933 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:14.843887    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:59:24.843369 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:24.843324    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:59:34.843519 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:34.843462    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:59:43.115095 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:43.115040    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 27 17:59:53.120249 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:53.120217    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 17:59:57.150333 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:57.150293    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"]
Jul 27 17:59:57.150995 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:57.150559    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container" containerID="cri-o://dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c" gracePeriod=30
Jul 27 17:59:57.150995 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:57.150881    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kube-rbac-proxy" containerID="cri-o://5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3" gracePeriod=30
Jul 27 17:59:58.217978 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:58.217776    2564 generic.go:356] "Generic (PLEG): container finished" podID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerID="5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3" exitCode=2
Jul 27 17:59:58.217978 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:58.217904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" event={"ID":"2c1fdec1-4233-48fe-bf05-22fbeb92decf","Type":"ContainerDied","Data":"5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3"}
Jul 27 17:59:59.838930 ip-10-0-129-58 kubenswrapper[2564]: I0727 17:59:59.838877    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 27 18:00:02.796157 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.796126    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 18:00:02.923343 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.923242    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-cabundle-cert\") pod \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") "
Jul 27 18:00:02.923343 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.923312    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-rbac-proxy-sar-config\") pod \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") "
Jul 27 18:00:02.923343 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.923341    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92gtt\" (UniqueName: \"kubernetes.io/projected/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-api-access-92gtt\") pod \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") "
Jul 27 18:00:02.923648 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.923382    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kserve-provision-location\") pod \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") "
Jul 27 18:00:02.923648 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.923410    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1fdec1-4233-48fe-bf05-22fbeb92decf-proxy-tls\") pod \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\" (UID: \"2c1fdec1-4233-48fe-bf05-22fbeb92decf\") "
Jul 27 18:00:02.923758 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.923687    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c1fdec1-4233-48fe-bf05-22fbeb92decf" (UID: "2c1fdec1-4233-48fe-bf05-22fbeb92decf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:00:02.923758 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.923733    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c1fdec1-4233-48fe-bf05-22fbeb92decf" (UID: "2c1fdec1-4233-48fe-bf05-22fbeb92decf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:00:02.923859 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.923802    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c1fdec1-4233-48fe-bf05-22fbeb92decf" (UID: "2c1fdec1-4233-48fe-bf05-22fbeb92decf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:00:02.925860 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.925827    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1fdec1-4233-48fe-bf05-22fbeb92decf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c1fdec1-4233-48fe-bf05-22fbeb92decf" (UID: "2c1fdec1-4233-48fe-bf05-22fbeb92decf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:00:02.925977 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:02.925856    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-api-access-92gtt" (OuterVolumeSpecName: "kube-api-access-92gtt") pod "2c1fdec1-4233-48fe-bf05-22fbeb92decf" (UID: "2c1fdec1-4233-48fe-bf05-22fbeb92decf"). InnerVolumeSpecName "kube-api-access-92gtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:00:03.024233 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.024153    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:00:03.024233 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.024225    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:00:03.024233 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.024236    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92gtt\" (UniqueName: \"kubernetes.io/projected/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kube-api-access-92gtt\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:00:03.024233 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.024245    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c1fdec1-4233-48fe-bf05-22fbeb92decf-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:00:03.024523 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.024256    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c1fdec1-4233-48fe-bf05-22fbeb92decf-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:00:03.241538 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.241497    2564 generic.go:356] "Generic (PLEG): container finished" podID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerID="dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c" exitCode=0
Jul 27 18:00:03.241744 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.241556    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" event={"ID":"2c1fdec1-4233-48fe-bf05-22fbeb92decf","Type":"ContainerDied","Data":"dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c"}
Jul 27 18:00:03.241744 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.241564    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"
Jul 27 18:00:03.241744 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.241605    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8" event={"ID":"2c1fdec1-4233-48fe-bf05-22fbeb92decf","Type":"ContainerDied","Data":"1a28d84c3802517ff07f77f5c3fa136cfeba184b8f99bd570c36afd506ebd904"}
Jul 27 18:00:03.241744 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.241635    2564 scope.go:117] "RemoveContainer" containerID="5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3"
Jul 27 18:00:03.250676 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.250653    2564 scope.go:117] "RemoveContainer" containerID="dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c"
Jul 27 18:00:03.259117 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.259095    2564 scope.go:117] "RemoveContainer" containerID="aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6"
Jul 27 18:00:03.264872 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.264842    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"]
Jul 27 18:00:03.267650 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.267625    2564 scope.go:117] "RemoveContainer" containerID="5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3"
Jul 27 18:00:03.268159 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:00:03.268134    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3\": container with ID starting with 5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3 not found: ID does not exist" containerID="5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3"
Jul 27 18:00:03.268359 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.268326    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3"} err="failed to get container status \"5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3\": rpc error: code = NotFound desc = could not find container \"5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3\": container with ID starting with 5bae24a624590c48041888b89c177a98a0a56bd0dc1f1f1de38b82459a06e3f3 not found: ID does not exist"
Jul 27 18:00:03.268487 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.268473    2564 scope.go:117] "RemoveContainer" containerID="dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c"
Jul 27 18:00:03.268881 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:00:03.268856    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c\": container with ID starting with dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c not found: ID does not exist" containerID="dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c"
Jul 27 18:00:03.268971 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.268887    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c"} err="failed to get container status \"dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c\": rpc error: code = NotFound desc = could not find container \"dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c\": container with ID starting with dd23433ab99076d1865c0e59ce430255263e4d54a70f0d6947da800c595dc84c not found: ID does not exist"
Jul 27 18:00:03.268971 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.268907    2564 scope.go:117] "RemoveContainer" containerID="aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6"
Jul 27 18:00:03.269259 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:00:03.269227    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6\": container with ID starting with aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6 not found: ID does not exist" containerID="aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6"
Jul 27 18:00:03.269342 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.269261    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6"} err="failed to get container status \"aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6\": rpc error: code = NotFound desc = could not find container \"aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6\": container with ID starting with aa47e374dcffb997db134d00c963ffcaaab63fd30837b4683fcc749784c6d3a6 not found: ID does not exist"
Jul 27 18:00:03.270364 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:03.270345    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gnfs8"]
Jul 27 18:00:05.120766 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:00:05.120726    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" path="/var/lib/kubelet/pods/2c1fdec1-4233-48fe-bf05-22fbeb92decf/volumes"
Jul 27 18:01:32.383454 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.383419    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"]
Jul 27 18:01:32.383966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.383781    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kube-rbac-proxy"
Jul 27 18:01:32.383966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.383794    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kube-rbac-proxy"
Jul 27 18:01:32.383966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.383815    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container"
Jul 27 18:01:32.383966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.383822    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container"
Jul 27 18:01:32.383966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.383836    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="storage-initializer"
Jul 27 18:01:32.383966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.383842    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="storage-initializer"
Jul 27 18:01:32.383966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.383916    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kserve-container"
Jul 27 18:01:32.383966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.383927    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1fdec1-4233-48fe-bf05-22fbeb92decf" containerName="kube-rbac-proxy"
Jul 27 18:01:32.386024 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.386005    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.389231 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.389202    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:01:32.389399 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.389247    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:01:32.389399 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.389310    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:01:32.390471 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.390453    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 27 18:01:32.390518 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.390504    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 27 18:01:32.390810 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.390792    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:01:32.390906 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.390892    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:01:32.396843 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.396814    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"]
Jul 27 18:01:32.418591 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.418532    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.418824 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.418669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45a94d9e-a876-4cf2-82b9-e77cf06cb143-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.418824 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.418699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bjv\" (UniqueName: \"kubernetes.io/projected/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-api-access-b2bjv\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.418824 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.418766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.418824 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.418807    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.519806 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.519671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45a94d9e-a876-4cf2-82b9-e77cf06cb143-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.519806 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.519725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bjv\" (UniqueName: \"kubernetes.io/projected/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-api-access-b2bjv\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.519806 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.519769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.519806 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.519791    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.525502 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.519826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.525502 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.520351    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.525502 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.520523    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.525502 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.520545    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.525502 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.522318    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45a94d9e-a876-4cf2-82b9-e77cf06cb143-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.527924 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.527900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bjv\" (UniqueName: \"kubernetes.io/projected/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-api-access-b2bjv\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.698135 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.698037    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:32.840846 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:32.840812    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"]
Jul 27 18:01:32.844308 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:01:32.844262    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a94d9e_a876_4cf2_82b9_e77cf06cb143.slice/crio-cb986792e8a92f172652b391cd898cc41f00bbacb542d76950cc72ae667c52a2 WatchSource:0}: Error finding container cb986792e8a92f172652b391cd898cc41f00bbacb542d76950cc72ae667c52a2: Status 404 returned error can't find the container with id cb986792e8a92f172652b391cd898cc41f00bbacb542d76950cc72ae667c52a2
Jul 27 18:01:33.622955 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:33.622915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" event={"ID":"45a94d9e-a876-4cf2-82b9-e77cf06cb143","Type":"ContainerStarted","Data":"9f10ae021e408b30b56dec599e778f3808e872b171c24262f14d31ebcd8480e5"}
Jul 27 18:01:33.623416 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:33.622985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" event={"ID":"45a94d9e-a876-4cf2-82b9-e77cf06cb143","Type":"ContainerStarted","Data":"cb986792e8a92f172652b391cd898cc41f00bbacb542d76950cc72ae667c52a2"}
Jul 27 18:01:36.637421 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:36.637385    2564 generic.go:356] "Generic (PLEG): container finished" podID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerID="9f10ae021e408b30b56dec599e778f3808e872b171c24262f14d31ebcd8480e5" exitCode=0
Jul 27 18:01:36.637822 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:36.637466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" event={"ID":"45a94d9e-a876-4cf2-82b9-e77cf06cb143","Type":"ContainerDied","Data":"9f10ae021e408b30b56dec599e778f3808e872b171c24262f14d31ebcd8480e5"}
Jul 27 18:01:37.643706 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:37.643668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" event={"ID":"45a94d9e-a876-4cf2-82b9-e77cf06cb143","Type":"ContainerStarted","Data":"7c80447b2c22ff025ffe4b01cba37235298ab937dc3571fc44c1a1550b2ffd21"}
Jul 27 18:01:37.644203 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:37.643764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" event={"ID":"45a94d9e-a876-4cf2-82b9-e77cf06cb143","Type":"ContainerStarted","Data":"4a5f0812d9dc32923715649c5728e7d94961abc04a95ba030781c7f7b19f2fb1"}
Jul 27 18:01:37.644203 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:37.644016    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:37.667857 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:37.667791    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podStartSLOduration=2.014935116 podStartE2EDuration="5.667774186s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.65283907s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:01:37.666332763 +0000 UTC m=+2041.118953912" watchObservedRunningTime="2026-07-27 18:01:37.667774186 +0000 UTC m=+2041.120395332"
Jul 27 18:01:38.648645 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:38.648605    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:38.650169 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:38.650136    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:01:39.653038 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:39.652992    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:01:44.658973 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:44.658939    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:01:44.659484 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:44.659453    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:01:54.660467 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:01:54.660405    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:02:04.660265 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:04.660136    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:02:14.659511 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:14.659460    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:02:24.659850 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:24.659801    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:02:34.659932 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:34.659881    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:02:37.085300 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:37.085269    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:02:37.089874 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:37.089845    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:02:37.093668 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:37.093642    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:02:37.101622 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:37.101593    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:02:44.659693 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:44.659647    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:02:51.120504 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:02:51.120443    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 27 18:03:01.121496 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:01.121460    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:03:02.489776 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:02.489727    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"]
Jul 27 18:03:02.490223 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:02.490034    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container" containerID="cri-o://4a5f0812d9dc32923715649c5728e7d94961abc04a95ba030781c7f7b19f2fb1" gracePeriod=30
Jul 27 18:03:02.490223 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:02.490116    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kube-rbac-proxy" containerID="cri-o://7c80447b2c22ff025ffe4b01cba37235298ab937dc3571fc44c1a1550b2ffd21" gracePeriod=30
Jul 27 18:03:03.019767 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:03.019731    2564 generic.go:356] "Generic (PLEG): container finished" podID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerID="7c80447b2c22ff025ffe4b01cba37235298ab937dc3571fc44c1a1550b2ffd21" exitCode=2
Jul 27 18:03:03.019962 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:03.019802    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" event={"ID":"45a94d9e-a876-4cf2-82b9-e77cf06cb143","Type":"ContainerDied","Data":"7c80447b2c22ff025ffe4b01cba37235298ab937dc3571fc44c1a1550b2ffd21"}
Jul 27 18:03:04.654063 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:04.654008    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 27 18:03:08.048762 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.048719    2564 generic.go:356] "Generic (PLEG): container finished" podID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerID="4a5f0812d9dc32923715649c5728e7d94961abc04a95ba030781c7f7b19f2fb1" exitCode=0
Jul 27 18:03:08.049197 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.048789    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" event={"ID":"45a94d9e-a876-4cf2-82b9-e77cf06cb143","Type":"ContainerDied","Data":"4a5f0812d9dc32923715649c5728e7d94961abc04a95ba030781c7f7b19f2fb1"}
Jul 27 18:03:08.154438 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.154367    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:03:08.205943 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.205901    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45a94d9e-a876-4cf2-82b9-e77cf06cb143-proxy-tls\") pod \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") "
Jul 27 18:03:08.206128 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.205992    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-cabundle-cert\") pod \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") "
Jul 27 18:03:08.206128 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.206028    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2bjv\" (UniqueName: \"kubernetes.io/projected/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-api-access-b2bjv\") pod \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") "
Jul 27 18:03:08.206128 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.206055    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-rbac-proxy-sar-config\") pod \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") "
Jul 27 18:03:08.206128 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.206099    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kserve-provision-location\") pod \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\" (UID: \"45a94d9e-a876-4cf2-82b9-e77cf06cb143\") "
Jul 27 18:03:08.206604 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.206477    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45a94d9e-a876-4cf2-82b9-e77cf06cb143" (UID: "45a94d9e-a876-4cf2-82b9-e77cf06cb143"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:03:08.206604 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.206518    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45a94d9e-a876-4cf2-82b9-e77cf06cb143" (UID: "45a94d9e-a876-4cf2-82b9-e77cf06cb143"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:03:08.206604 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.206549    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45a94d9e-a876-4cf2-82b9-e77cf06cb143" (UID: "45a94d9e-a876-4cf2-82b9-e77cf06cb143"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:03:08.208356 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.208334    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a94d9e-a876-4cf2-82b9-e77cf06cb143-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45a94d9e-a876-4cf2-82b9-e77cf06cb143" (UID: "45a94d9e-a876-4cf2-82b9-e77cf06cb143"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:03:08.208452 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.208430    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-api-access-b2bjv" (OuterVolumeSpecName: "kube-api-access-b2bjv") pod "45a94d9e-a876-4cf2-82b9-e77cf06cb143" (UID: "45a94d9e-a876-4cf2-82b9-e77cf06cb143"). InnerVolumeSpecName "kube-api-access-b2bjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:03:08.307631 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.307580    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:03:08.307631 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.307627    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2bjv\" (UniqueName: \"kubernetes.io/projected/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-api-access-b2bjv\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:03:08.307631 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.307640    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:03:08.307631 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.307650    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45a94d9e-a876-4cf2-82b9-e77cf06cb143-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:03:08.307931 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:08.307659    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45a94d9e-a876-4cf2-82b9-e77cf06cb143-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:03:09.055496 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:09.055460    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz" event={"ID":"45a94d9e-a876-4cf2-82b9-e77cf06cb143","Type":"ContainerDied","Data":"cb986792e8a92f172652b391cd898cc41f00bbacb542d76950cc72ae667c52a2"}
Jul 27 18:03:09.055929 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:09.055532    2564 scope.go:117] "RemoveContainer" containerID="7c80447b2c22ff025ffe4b01cba37235298ab937dc3571fc44c1a1550b2ffd21"
Jul 27 18:03:09.055929 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:09.055467    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"
Jul 27 18:03:09.065270 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:09.065249    2564 scope.go:117] "RemoveContainer" containerID="4a5f0812d9dc32923715649c5728e7d94961abc04a95ba030781c7f7b19f2fb1"
Jul 27 18:03:09.077105 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:09.076679    2564 scope.go:117] "RemoveContainer" containerID="9f10ae021e408b30b56dec599e778f3808e872b171c24262f14d31ebcd8480e5"
Jul 27 18:03:09.079349 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:09.079324    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"]
Jul 27 18:03:09.086006 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:09.085976    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-z64gz"]
Jul 27 18:03:09.120323 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:03:09.120289    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" path="/var/lib/kubelet/pods/45a94d9e-a876-4cf2-82b9-e77cf06cb143/volumes"
Jul 27 18:07:37.111735 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:07:37.111704    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:07:37.123526 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:07:37.123497    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:07:37.125719 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:07:37.125701    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:07:37.133555 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:07:37.133538    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:12:37.141602 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:12:37.141532    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:12:37.152318 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:12:37.152293    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:12:37.155025 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:12:37.155007    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:12:37.162072 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:12:37.162054    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:14:09.865476 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.865385    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"]
Jul 27 18:14:09.866067 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.865843    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container"
Jul 27 18:14:09.866067 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.865861    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container"
Jul 27 18:14:09.866067 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.865880    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kube-rbac-proxy"
Jul 27 18:14:09.866067 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.865889    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kube-rbac-proxy"
Jul 27 18:14:09.866067 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.865905    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="storage-initializer"
Jul 27 18:14:09.866067 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.865915    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="storage-initializer"
Jul 27 18:14:09.866067 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.866000    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kube-rbac-proxy"
Jul 27 18:14:09.866067 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.866015    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45a94d9e-a876-4cf2-82b9-e77cf06cb143" containerName="kserve-container"
Jul 27 18:14:09.869562 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.869542    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:09.873915 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.873895    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 27 18:14:09.875813 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.875795    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:14:09.875919 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.875873    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:14:09.876277 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.876252    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:14:09.876880 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.876857    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 27 18:14:09.877095 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.877078    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:14:09.879968 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.879949    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:14:09.884574 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.884550    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"]
Jul 27 18:14:09.919783 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.919746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltwf\" (UniqueName: \"kubernetes.io/projected/bd01a7be-a0df-43df-a691-30ee9851212b-kube-api-access-dltwf\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:09.919783 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.919784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:09.919984 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.919836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd01a7be-a0df-43df-a691-30ee9851212b-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:09.919984 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.919878    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd01a7be-a0df-43df-a691-30ee9851212b-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:09.919984 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:09.919961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.020779 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.020739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltwf\" (UniqueName: \"kubernetes.io/projected/bd01a7be-a0df-43df-a691-30ee9851212b-kube-api-access-dltwf\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.020779 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.020780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.021047 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.020803    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd01a7be-a0df-43df-a691-30ee9851212b-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.021047 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.020830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd01a7be-a0df-43df-a691-30ee9851212b-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.021047 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.020886    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.021365 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.021343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd01a7be-a0df-43df-a691-30ee9851212b-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.021489 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.021469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.021599 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.021583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.023304 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.023271    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd01a7be-a0df-43df-a691-30ee9851212b-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.032769 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.032744    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltwf\" (UniqueName: \"kubernetes.io/projected/bd01a7be-a0df-43df-a691-30ee9851212b-kube-api-access-dltwf\") pod \"isvc-paddle-predictor-6d5c9fd4f6-xfd74\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.182613 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.182525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:10.315706 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.315670    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"]
Jul 27 18:14:10.318742 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:14:10.318707    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd01a7be_a0df_43df_a691_30ee9851212b.slice/crio-d9fc1a868788a705d326d1f5dc2416da359687aa70ad01df3b5d69e8bd7b1f4a WatchSource:0}: Error finding container d9fc1a868788a705d326d1f5dc2416da359687aa70ad01df3b5d69e8bd7b1f4a: Status 404 returned error can't find the container with id d9fc1a868788a705d326d1f5dc2416da359687aa70ad01df3b5d69e8bd7b1f4a
Jul 27 18:14:10.713456 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.713412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" event={"ID":"bd01a7be-a0df-43df-a691-30ee9851212b","Type":"ContainerStarted","Data":"d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9"}
Jul 27 18:14:10.713643 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:10.713487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" event={"ID":"bd01a7be-a0df-43df-a691-30ee9851212b","Type":"ContainerStarted","Data":"d9fc1a868788a705d326d1f5dc2416da359687aa70ad01df3b5d69e8bd7b1f4a"}
Jul 27 18:14:15.734590 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:15.734550    2564 generic.go:356] "Generic (PLEG): container finished" podID="bd01a7be-a0df-43df-a691-30ee9851212b" containerID="d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9" exitCode=0
Jul 27 18:14:15.734979 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:15.734606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" event={"ID":"bd01a7be-a0df-43df-a691-30ee9851212b","Type":"ContainerDied","Data":"d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9"}
Jul 27 18:14:15.736020 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:15.736004    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:14:29.812352 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:29.812315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" event={"ID":"bd01a7be-a0df-43df-a691-30ee9851212b","Type":"ContainerStarted","Data":"6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50"}
Jul 27 18:14:29.812838 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:29.812394    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" event={"ID":"bd01a7be-a0df-43df-a691-30ee9851212b","Type":"ContainerStarted","Data":"a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352"}
Jul 27 18:14:29.812838 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:29.812691    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:29.812838 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:29.812732    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:29.813942 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:29.813914    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 27 18:14:29.839119 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:29.839065    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" podStartSLOduration=2.447551938 podStartE2EDuration="20.839049402s" totalImagesPullingTime="13.903592861s" totalInitContainerRuntime="4.487904603s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:14:29.835756128 +0000 UTC m=+2813.288377273" watchObservedRunningTime="2026-07-27 18:14:29.839049402 +0000 UTC m=+2813.291670548"
Jul 27 18:14:30.816850 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:30.816810    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 27 18:14:35.821709 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:35.821675    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:14:35.822402 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:35.822367    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 27 18:14:45.822460 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:45.822423    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 27 18:14:55.823124 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:14:55.823066    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 27 18:15:05.822252 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:05.822217    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 27 18:15:15.822913 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:15.822883    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:15:21.361845 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:21.361807    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"]
Jul 27 18:15:21.362404 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:21.362099    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container" containerID="cri-o://a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352" gracePeriod=30
Jul 27 18:15:21.362404 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:21.362172    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kube-rbac-proxy" containerID="cri-o://6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50" gracePeriod=30
Jul 27 18:15:22.029724 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:22.029686    2564 generic.go:356] "Generic (PLEG): container finished" podID="bd01a7be-a0df-43df-a691-30ee9851212b" containerID="6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50" exitCode=2
Jul 27 18:15:22.029893 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:22.029736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" event={"ID":"bd01a7be-a0df-43df-a691-30ee9851212b","Type":"ContainerDied","Data":"6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50"}
Jul 27 18:15:24.005368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.005346    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:15:24.040810 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.040778    2564 generic.go:356] "Generic (PLEG): container finished" podID="bd01a7be-a0df-43df-a691-30ee9851212b" containerID="a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352" exitCode=0
Jul 27 18:15:24.040981 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.040828    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" event={"ID":"bd01a7be-a0df-43df-a691-30ee9851212b","Type":"ContainerDied","Data":"a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352"}
Jul 27 18:15:24.040981 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.040875    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"
Jul 27 18:15:24.040981 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.040915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74" event={"ID":"bd01a7be-a0df-43df-a691-30ee9851212b","Type":"ContainerDied","Data":"d9fc1a868788a705d326d1f5dc2416da359687aa70ad01df3b5d69e8bd7b1f4a"}
Jul 27 18:15:24.040981 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.040954    2564 scope.go:117] "RemoveContainer" containerID="6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50"
Jul 27 18:15:24.049397 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.049373    2564 scope.go:117] "RemoveContainer" containerID="a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352"
Jul 27 18:15:24.056699 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.056677    2564 scope.go:117] "RemoveContainer" containerID="d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9"
Jul 27 18:15:24.063901 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.063884    2564 scope.go:117] "RemoveContainer" containerID="6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50"
Jul 27 18:15:24.064201 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:15:24.064156    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50\": container with ID starting with 6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50 not found: ID does not exist" containerID="6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50"
Jul 27 18:15:24.064273 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.064216    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50"} err="failed to get container status \"6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50\": rpc error: code = NotFound desc = could not find container \"6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50\": container with ID starting with 6c71faa17f74af9c70680c218b02d06127cb07b88379787e2777d5b58c944a50 not found: ID does not exist"
Jul 27 18:15:24.064273 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.064243    2564 scope.go:117] "RemoveContainer" containerID="a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352"
Jul 27 18:15:24.064506 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:15:24.064486    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352\": container with ID starting with a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352 not found: ID does not exist" containerID="a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352"
Jul 27 18:15:24.064551 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.064512    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352"} err="failed to get container status \"a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352\": rpc error: code = NotFound desc = could not find container \"a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352\": container with ID starting with a8c6bef8e0a42d958692c91fa1f7587b45d17a8703accb31f70f95b51c40f352 not found: ID does not exist"
Jul 27 18:15:24.064551 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.064529    2564 scope.go:117] "RemoveContainer" containerID="d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9"
Jul 27 18:15:24.064770 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:15:24.064755    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9\": container with ID starting with d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9 not found: ID does not exist" containerID="d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9"
Jul 27 18:15:24.064818 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.064772    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9"} err="failed to get container status \"d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9\": rpc error: code = NotFound desc = could not find container \"d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9\": container with ID starting with d7095351a742a5884764a8ab29228d1b575f4773f653aec876e149829c8ba4f9 not found: ID does not exist"
Jul 27 18:15:24.172769 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.172735    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-kube-rbac-proxy-sar-config\") pod \"bd01a7be-a0df-43df-a691-30ee9851212b\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") "
Jul 27 18:15:24.172977 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.172788    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-cabundle-cert\") pod \"bd01a7be-a0df-43df-a691-30ee9851212b\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") "
Jul 27 18:15:24.172977 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.172808    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd01a7be-a0df-43df-a691-30ee9851212b-proxy-tls\") pod \"bd01a7be-a0df-43df-a691-30ee9851212b\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") "
Jul 27 18:15:24.172977 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.172837    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd01a7be-a0df-43df-a691-30ee9851212b-kserve-provision-location\") pod \"bd01a7be-a0df-43df-a691-30ee9851212b\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") "
Jul 27 18:15:24.172977 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.172907    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dltwf\" (UniqueName: \"kubernetes.io/projected/bd01a7be-a0df-43df-a691-30ee9851212b-kube-api-access-dltwf\") pod \"bd01a7be-a0df-43df-a691-30ee9851212b\" (UID: \"bd01a7be-a0df-43df-a691-30ee9851212b\") "
Jul 27 18:15:24.173311 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.173233    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd01a7be-a0df-43df-a691-30ee9851212b" (UID: "bd01a7be-a0df-43df-a691-30ee9851212b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:15:24.173409 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.173385    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd01a7be-a0df-43df-a691-30ee9851212b" (UID: "bd01a7be-a0df-43df-a691-30ee9851212b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:15:24.174874 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.174846    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd01a7be-a0df-43df-a691-30ee9851212b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd01a7be-a0df-43df-a691-30ee9851212b" (UID: "bd01a7be-a0df-43df-a691-30ee9851212b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:15:24.174997 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.174976    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd01a7be-a0df-43df-a691-30ee9851212b-kube-api-access-dltwf" (OuterVolumeSpecName: "kube-api-access-dltwf") pod "bd01a7be-a0df-43df-a691-30ee9851212b" (UID: "bd01a7be-a0df-43df-a691-30ee9851212b"). InnerVolumeSpecName "kube-api-access-dltwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:15:24.181774 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.181746    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd01a7be-a0df-43df-a691-30ee9851212b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd01a7be-a0df-43df-a691-30ee9851212b" (UID: "bd01a7be-a0df-43df-a691-30ee9851212b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:15:24.274192 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.274156    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:15:24.274324 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.274209    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd01a7be-a0df-43df-a691-30ee9851212b-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:15:24.274324 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.274224    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd01a7be-a0df-43df-a691-30ee9851212b-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:15:24.274324 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.274236    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd01a7be-a0df-43df-a691-30ee9851212b-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:15:24.274324 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.274249    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dltwf\" (UniqueName: \"kubernetes.io/projected/bd01a7be-a0df-43df-a691-30ee9851212b-kube-api-access-dltwf\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:15:24.367707 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.367674    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"]
Jul 27 18:15:24.374551 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:24.374526    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-xfd74"]
Jul 27 18:15:25.119465 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:15:25.119434    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" path="/var/lib/kubelet/pods/bd01a7be-a0df-43df-a691-30ee9851212b/volumes"
Jul 27 18:17:37.171340 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:17:37.171314    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:17:37.179321 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:17:37.179298    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:17:37.182417 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:17:37.182401    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:17:37.189500 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:17:37.189484    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:22:37.198880 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:22:37.198851    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:22:37.206573 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:22:37.206551    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:22:37.208758 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:22:37.208740    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:22:37.215782 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:22:37.215765    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:27:37.225161 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:27:37.225134    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:27:37.233326 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:27:37.233304    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:27:37.235765 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:27:37.235747    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:27:37.243127 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:27:37.243110    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:32:37.250366 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:32:37.250263    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:32:37.258431 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:32:37.258411    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:32:37.262438 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:32:37.262422    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:32:37.269389 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:32:37.269373    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:37:37.275666 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:37:37.275552    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:37:37.290108 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:37:37.290086    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:37:37.294643 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:37:37.294623    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:37:37.301657 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:37:37.301638    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:38:13.177527 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.177448    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"]
Jul 27 18:38:13.177979 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.177743    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kube-rbac-proxy"
Jul 27 18:38:13.177979 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.177754    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kube-rbac-proxy"
Jul 27 18:38:13.177979 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.177767    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container"
Jul 27 18:38:13.177979 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.177773    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container"
Jul 27 18:38:13.177979 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.177780    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="storage-initializer"
Jul 27 18:38:13.177979 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.177785    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="storage-initializer"
Jul 27 18:38:13.177979 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.177843    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kube-rbac-proxy"
Jul 27 18:38:13.177979 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.177850    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd01a7be-a0df-43df-a691-30ee9851212b" containerName="kserve-container"
Jul 27 18:38:13.180906 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.180889    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.183674 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.183649    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:38:13.183795 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.183703    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 27 18:38:13.183913 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.183657    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:38:13.183981 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.183920    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:38:13.183981 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.183949    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:38:13.184753 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.184737    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:38:13.184852 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.184754    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:38:13.192457 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.192432    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"]
Jul 27 18:38:13.280367 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.280337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c8fafc-2764-4cab-894c-e600ab72db4d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.280367 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.280370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.280649 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.280394    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.280649 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.280449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c8fafc-2764-4cab-894c-e600ab72db4d-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.280649 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.280566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpw2t\" (UniqueName: \"kubernetes.io/projected/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-api-access-kpw2t\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.381567 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.381529    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.381567 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.381568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c8fafc-2764-4cab-894c-e600ab72db4d-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.381810 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.381693    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpw2t\" (UniqueName: \"kubernetes.io/projected/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-api-access-kpw2t\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.381810 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.381781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c8fafc-2764-4cab-894c-e600ab72db4d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.381810 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.381801    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.381962 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.381943    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c8fafc-2764-4cab-894c-e600ab72db4d-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.382292 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.382265    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.382397 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.382348    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.384119 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.384099    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c8fafc-2764-4cab-894c-e600ab72db4d-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.390121 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.390102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpw2t\" (UniqueName: \"kubernetes.io/projected/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-api-access-kpw2t\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-7d5vv\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.491613 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.491566    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:13.622992 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:13.622948    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"]
Jul 27 18:38:13.625141 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:38:13.625105    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c8fafc_2764_4cab_894c_e600ab72db4d.slice/crio-dfdc0496022b5faf5745d8ca1850909719b6d90504b4aac148899b88099c5854 WatchSource:0}: Error finding container dfdc0496022b5faf5745d8ca1850909719b6d90504b4aac148899b88099c5854: Status 404 returned error can't find the container with id dfdc0496022b5faf5745d8ca1850909719b6d90504b4aac148899b88099c5854
Jul 27 18:38:14.528551 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:14.528517    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" event={"ID":"43c8fafc-2764-4cab-894c-e600ab72db4d","Type":"ContainerStarted","Data":"539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070"}
Jul 27 18:38:14.528934 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:14.528588    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" event={"ID":"43c8fafc-2764-4cab-894c-e600ab72db4d","Type":"ContainerStarted","Data":"dfdc0496022b5faf5745d8ca1850909719b6d90504b4aac148899b88099c5854"}
Jul 27 18:38:19.554149 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:19.554113    2564 generic.go:356] "Generic (PLEG): container finished" podID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerID="539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070" exitCode=0
Jul 27 18:38:19.554695 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:19.554207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" event={"ID":"43c8fafc-2764-4cab-894c-e600ab72db4d","Type":"ContainerDied","Data":"539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070"}
Jul 27 18:38:20.561352 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:20.561314    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" event={"ID":"43c8fafc-2764-4cab-894c-e600ab72db4d","Type":"ContainerStarted","Data":"6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb"}
Jul 27 18:38:20.561784 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:20.561380    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" event={"ID":"43c8fafc-2764-4cab-894c-e600ab72db4d","Type":"ContainerStarted","Data":"ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f"}
Jul 27 18:38:20.561784 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:20.561578    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:20.583902 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:20.583848    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" podStartSLOduration=2.498436952 podStartE2EDuration="7.583830191s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.085393239s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:20.582751543 +0000 UTC m=+4244.035372725" watchObservedRunningTime="2026-07-27 18:38:20.583830191 +0000 UTC m=+4244.036451335"
Jul 27 18:38:21.565835 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:21.565788    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:21.567157 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:21.567126    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 27 18:38:22.569607 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:22.569567    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 27 18:38:27.573900 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:27.573870    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:27.574459 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:27.574430    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 27 18:38:37.574713 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:37.574679    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:44.914582 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:44.914553    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-599b58cf6-7d5vv_43c8fafc-2764-4cab-894c-e600ab72db4d/kserve-container/0.log"
Jul 27 18:38:45.058059 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:45.058028    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"]
Jul 27 18:38:45.058331 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:45.058308    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kserve-container" containerID="cri-o://ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f" gracePeriod=30
Jul 27 18:38:45.058413 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:45.058329    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kube-rbac-proxy" containerID="cri-o://6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb" gracePeriod=30
Jul 27 18:38:45.670657 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:45.670624    2564 generic.go:356] "Generic (PLEG): container finished" podID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerID="6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb" exitCode=2
Jul 27 18:38:45.670830 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:45.670698    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" event={"ID":"43c8fafc-2764-4cab-894c-e600ab72db4d","Type":"ContainerDied","Data":"6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb"}
Jul 27 18:38:46.298733 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.298713    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:46.473810 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.473788    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c8fafc-2764-4cab-894c-e600ab72db4d-kserve-provision-location\") pod \"43c8fafc-2764-4cab-894c-e600ab72db4d\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") "
Jul 27 18:38:46.473966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.473840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-rbac-proxy-sar-config\") pod \"43c8fafc-2764-4cab-894c-e600ab72db4d\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") "
Jul 27 18:38:46.473966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.473858    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpw2t\" (UniqueName: \"kubernetes.io/projected/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-api-access-kpw2t\") pod \"43c8fafc-2764-4cab-894c-e600ab72db4d\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") "
Jul 27 18:38:46.473966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.473892    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c8fafc-2764-4cab-894c-e600ab72db4d-proxy-tls\") pod \"43c8fafc-2764-4cab-894c-e600ab72db4d\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") "
Jul 27 18:38:46.473966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.473929    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-cabundle-cert\") pod \"43c8fafc-2764-4cab-894c-e600ab72db4d\" (UID: \"43c8fafc-2764-4cab-894c-e600ab72db4d\") "
Jul 27 18:38:46.474253 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.474219    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43c8fafc-2764-4cab-894c-e600ab72db4d" (UID: "43c8fafc-2764-4cab-894c-e600ab72db4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:38:46.474373 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.474346    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43c8fafc-2764-4cab-894c-e600ab72db4d" (UID: "43c8fafc-2764-4cab-894c-e600ab72db4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:38:46.475967 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.475945    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c8fafc-2764-4cab-894c-e600ab72db4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43c8fafc-2764-4cab-894c-e600ab72db4d" (UID: "43c8fafc-2764-4cab-894c-e600ab72db4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:38:46.476056 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.476013    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-api-access-kpw2t" (OuterVolumeSpecName: "kube-api-access-kpw2t") pod "43c8fafc-2764-4cab-894c-e600ab72db4d" (UID: "43c8fafc-2764-4cab-894c-e600ab72db4d"). InnerVolumeSpecName "kube-api-access-kpw2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:38:46.497989 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.497964    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c8fafc-2764-4cab-894c-e600ab72db4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43c8fafc-2764-4cab-894c-e600ab72db4d" (UID: "43c8fafc-2764-4cab-894c-e600ab72db4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:38:46.574618 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.574590    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:38:46.574618 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.574614    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpw2t\" (UniqueName: \"kubernetes.io/projected/43c8fafc-2764-4cab-894c-e600ab72db4d-kube-api-access-kpw2t\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:38:46.574771 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.574630    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c8fafc-2764-4cab-894c-e600ab72db4d-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:38:46.574771 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.574643    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c8fafc-2764-4cab-894c-e600ab72db4d-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:38:46.574771 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.574653    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c8fafc-2764-4cab-894c-e600ab72db4d-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:38:46.676521 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.676488    2564 generic.go:356] "Generic (PLEG): container finished" podID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerID="ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f" exitCode=0
Jul 27 18:38:46.676646 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.676554    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"
Jul 27 18:38:46.676646 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.676571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" event={"ID":"43c8fafc-2764-4cab-894c-e600ab72db4d","Type":"ContainerDied","Data":"ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f"}
Jul 27 18:38:46.676646 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.676641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv" event={"ID":"43c8fafc-2764-4cab-894c-e600ab72db4d","Type":"ContainerDied","Data":"dfdc0496022b5faf5745d8ca1850909719b6d90504b4aac148899b88099c5854"}
Jul 27 18:38:46.676771 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.676681    2564 scope.go:117] "RemoveContainer" containerID="6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb"
Jul 27 18:38:46.685262 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.685243    2564 scope.go:117] "RemoveContainer" containerID="ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f"
Jul 27 18:38:46.692545 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.692527    2564 scope.go:117] "RemoveContainer" containerID="539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070"
Jul 27 18:38:46.699388 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.699367    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"]
Jul 27 18:38:46.699669 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.699650    2564 scope.go:117] "RemoveContainer" containerID="6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb"
Jul 27 18:38:46.699925 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:38:46.699904    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb\": container with ID starting with 6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb not found: ID does not exist" containerID="6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb"
Jul 27 18:38:46.700004 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.699930    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb"} err="failed to get container status \"6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb\": rpc error: code = NotFound desc = could not find container \"6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb\": container with ID starting with 6a6d6aecf76930d7054d303d2f90b9433000b70d0f2b7429bfa518b0635369bb not found: ID does not exist"
Jul 27 18:38:46.700004 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.699947    2564 scope.go:117] "RemoveContainer" containerID="ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f"
Jul 27 18:38:46.700209 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:38:46.700168    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f\": container with ID starting with ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f not found: ID does not exist" containerID="ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f"
Jul 27 18:38:46.700272 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.700215    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f"} err="failed to get container status \"ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f\": rpc error: code = NotFound desc = could not find container \"ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f\": container with ID starting with ae63f5c005598ef497deeea80f067b23ebc537959e33cc0b6dc9cb6924808a3f not found: ID does not exist"
Jul 27 18:38:46.700272 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.700233    2564 scope.go:117] "RemoveContainer" containerID="539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070"
Jul 27 18:38:46.700443 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:38:46.700425    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070\": container with ID starting with 539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070 not found: ID does not exist" containerID="539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070"
Jul 27 18:38:46.700487 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.700449    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070"} err="failed to get container status \"539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070\": rpc error: code = NotFound desc = could not find container \"539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070\": container with ID starting with 539503cf51c9fdb97b3cf9a84d89c93063670cfb17ae8b13c896da3e9fe84070 not found: ID does not exist"
Jul 27 18:38:46.705121 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:46.705100    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-7d5vv"]
Jul 27 18:38:47.120221 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:38:47.120164    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" path="/var/lib/kubelet/pods/43c8fafc-2764-4cab-894c-e600ab72db4d/volumes"
Jul 27 18:42:37.309081 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:42:37.308985    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:42:37.317438 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:42:37.317419    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:42:37.321089 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:42:37.321067    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:42:37.328902 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:42:37.328884    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:43:52.711923 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.711883    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"]
Jul 27 18:43:52.712367 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.712246    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="storage-initializer"
Jul 27 18:43:52.712367 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.712266    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="storage-initializer"
Jul 27 18:43:52.712367 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.712297    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kube-rbac-proxy"
Jul 27 18:43:52.712367 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.712315    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kube-rbac-proxy"
Jul 27 18:43:52.712367 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.712331    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kserve-container"
Jul 27 18:43:52.712367 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.712337    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kserve-container"
Jul 27 18:43:52.712630 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.712401    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kserve-container"
Jul 27 18:43:52.712630 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.712411    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c8fafc-2764-4cab-894c-e600ab72db4d" containerName="kube-rbac-proxy"
Jul 27 18:43:52.715576 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.715558    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.718304 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.718279    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:43:52.718442 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.718305    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:43:52.718442 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.718348    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:43:52.718442 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.718306    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:43:52.718624 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.718451    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 27 18:43:52.718624 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.718511    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:43:52.719103 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.719075    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:43:52.729079 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.728937    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"]
Jul 27 18:43:52.846082 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.846051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.846242 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.846086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d4700b-30b9-4570-bbab-ca7c9c965b3f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.846242 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.846107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.846242 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.846132    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qlnk\" (UniqueName: \"kubernetes.io/projected/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-api-access-6qlnk\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.846242 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.846234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.947166 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.947137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.947314 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.947174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d4700b-30b9-4570-bbab-ca7c9c965b3f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.947314 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.947294    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.947425 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.947331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlnk\" (UniqueName: \"kubernetes.io/projected/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-api-access-6qlnk\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.947425 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.947397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.947815 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.947784    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.947919 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.947877    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.947984 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.947949    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.949557 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.949540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d4700b-30b9-4570-bbab-ca7c9c965b3f-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:52.956035 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:52.956012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlnk\" (UniqueName: \"kubernetes.io/projected/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-api-access-6qlnk\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:53.030923 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:53.030905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:43:53.163639 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:53.163611    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"]
Jul 27 18:43:53.165986 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:43:53.165960    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d4700b_30b9_4570_bbab_ca7c9c965b3f.slice/crio-c86f757c68cff94f81c5229215f0453268faf3c9d701fbf1824162bca04f294e WatchSource:0}: Error finding container c86f757c68cff94f81c5229215f0453268faf3c9d701fbf1824162bca04f294e: Status 404 returned error can't find the container with id c86f757c68cff94f81c5229215f0453268faf3c9d701fbf1824162bca04f294e
Jul 27 18:43:53.911549 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:53.911510    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" event={"ID":"67d4700b-30b9-4570-bbab-ca7c9c965b3f","Type":"ContainerStarted","Data":"109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472"}
Jul 27 18:43:53.911949 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:53.911581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" event={"ID":"67d4700b-30b9-4570-bbab-ca7c9c965b3f","Type":"ContainerStarted","Data":"c86f757c68cff94f81c5229215f0453268faf3c9d701fbf1824162bca04f294e"}
Jul 27 18:43:57.927686 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:57.927654    2564 generic.go:356] "Generic (PLEG): container finished" podID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerID="109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472" exitCode=0
Jul 27 18:43:57.927992 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:57.927705    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" event={"ID":"67d4700b-30b9-4570-bbab-ca7c9c965b3f","Type":"ContainerDied","Data":"109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472"}
Jul 27 18:43:57.928878 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:43:57.928862    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:44:01.948879 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:01.948798    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" event={"ID":"67d4700b-30b9-4570-bbab-ca7c9c965b3f","Type":"ContainerStarted","Data":"7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5"}
Jul 27 18:44:01.949271 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:01.948891    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" event={"ID":"67d4700b-30b9-4570-bbab-ca7c9c965b3f","Type":"ContainerStarted","Data":"0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58"}
Jul 27 18:44:01.949271 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:01.949084    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:44:01.972817 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:01.972773    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podStartSLOduration=1.692235854 podStartE2EDuration="9.972759996s" totalImagesPullingTime="3.631616308s" totalInitContainerRuntime="4.648907834s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:44:01.970105466 +0000 UTC m=+4585.422726612" watchObservedRunningTime="2026-07-27 18:44:01.972759996 +0000 UTC m=+4585.425381142"
Jul 27 18:44:02.953535 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:02.953504    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:44:02.954697 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:02.954665    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 27 18:44:03.957411 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:03.957375    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 27 18:44:08.961922 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:08.961884    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:44:08.962376 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:08.962339    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 27 18:44:18.963559 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:18.963530    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:44:32.547799 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:32.547762    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"]
Jul 27 18:44:32.548263 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:32.548020    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kserve-container" containerID="cri-o://0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58" gracePeriod=30
Jul 27 18:44:32.548263 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:32.548051    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy" containerID="cri-o://7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5" gracePeriod=30
Jul 27 18:44:33.084722 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:33.084689    2564 generic.go:356] "Generic (PLEG): container finished" podID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerID="7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5" exitCode=2
Jul 27 18:44:33.084928 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:33.084764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" event={"ID":"67d4700b-30b9-4570-bbab-ca7c9c965b3f","Type":"ContainerDied","Data":"7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5"}
Jul 27 18:44:33.957722 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:33.957671    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 27 18:44:38.958286 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:38.958237    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 27 18:44:43.957741 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:43.957694    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 27 18:44:43.958162 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:43.957821    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:44:48.958365 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:48.958322    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 27 18:44:53.958194 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:53.958134    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 27 18:44:58.958199 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:44:58.958144    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 27 18:45:03.190456 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.190430    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:45:03.211949 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.211915    2564 generic.go:356] "Generic (PLEG): container finished" podID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerID="0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58" exitCode=137
Jul 27 18:45:03.212086 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.211960    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" event={"ID":"67d4700b-30b9-4570-bbab-ca7c9c965b3f","Type":"ContainerDied","Data":"0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58"}
Jul 27 18:45:03.212086 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.211977    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"
Jul 27 18:45:03.212086 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.212017    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb" event={"ID":"67d4700b-30b9-4570-bbab-ca7c9c965b3f","Type":"ContainerDied","Data":"c86f757c68cff94f81c5229215f0453268faf3c9d701fbf1824162bca04f294e"}
Jul 27 18:45:03.212086 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.212056    2564 scope.go:117] "RemoveContainer" containerID="7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5"
Jul 27 18:45:03.220567 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.220542    2564 scope.go:117] "RemoveContainer" containerID="0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58"
Jul 27 18:45:03.228093 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.228069    2564 scope.go:117] "RemoveContainer" containerID="109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472"
Jul 27 18:45:03.235707 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.235685    2564 scope.go:117] "RemoveContainer" containerID="7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5"
Jul 27 18:45:03.235966 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:45:03.235949    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5\": container with ID starting with 7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5 not found: ID does not exist" containerID="7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5"
Jul 27 18:45:03.236024 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.235979    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5"} err="failed to get container status \"7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5\": rpc error: code = NotFound desc = could not find container \"7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5\": container with ID starting with 7d4295b52f86451fa7eeeedb70686b238da246273b98a79a5f8a3242e4196bc5 not found: ID does not exist"
Jul 27 18:45:03.236024 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.235999    2564 scope.go:117] "RemoveContainer" containerID="0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58"
Jul 27 18:45:03.236287 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:45:03.236267    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58\": container with ID starting with 0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58 not found: ID does not exist" containerID="0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58"
Jul 27 18:45:03.236361 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.236292    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58"} err="failed to get container status \"0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58\": rpc error: code = NotFound desc = could not find container \"0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58\": container with ID starting with 0e2670e3866607b6ff16940448cf4b775ee0bfb64b11f9f167c953a953bbcb58 not found: ID does not exist"
Jul 27 18:45:03.236361 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.236307    2564 scope.go:117] "RemoveContainer" containerID="109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472"
Jul 27 18:45:03.236587 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:45:03.236566    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472\": container with ID starting with 109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472 not found: ID does not exist" containerID="109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472"
Jul 27 18:45:03.236634 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.236596    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472"} err="failed to get container status \"109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472\": rpc error: code = NotFound desc = could not find container \"109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472\": container with ID starting with 109a4a18b5e7839eb04713399c247f6d50f31ef757bc7bb80c2e06cc47a13472 not found: ID does not exist"
Jul 27 18:45:03.305358 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.305326    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kserve-provision-location\") pod \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") "
Jul 27 18:45:03.305532 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.305375    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-cabundle-cert\") pod \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") "
Jul 27 18:45:03.305532 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.305400    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-rbac-proxy-sar-config\") pod \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") "
Jul 27 18:45:03.305532 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.305425    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d4700b-30b9-4570-bbab-ca7c9c965b3f-proxy-tls\") pod \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") "
Jul 27 18:45:03.305532 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.305442    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qlnk\" (UniqueName: \"kubernetes.io/projected/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-api-access-6qlnk\") pod \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\" (UID: \"67d4700b-30b9-4570-bbab-ca7c9c965b3f\") "
Jul 27 18:45:03.305804 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.305780    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67d4700b-30b9-4570-bbab-ca7c9c965b3f" (UID: "67d4700b-30b9-4570-bbab-ca7c9c965b3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:45:03.305895 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.305771    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67d4700b-30b9-4570-bbab-ca7c9c965b3f" (UID: "67d4700b-30b9-4570-bbab-ca7c9c965b3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:45:03.307657 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.307621    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d4700b-30b9-4570-bbab-ca7c9c965b3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67d4700b-30b9-4570-bbab-ca7c9c965b3f" (UID: "67d4700b-30b9-4570-bbab-ca7c9c965b3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:45:03.307657 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.307636    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-api-access-6qlnk" (OuterVolumeSpecName: "kube-api-access-6qlnk") pod "67d4700b-30b9-4570-bbab-ca7c9c965b3f" (UID: "67d4700b-30b9-4570-bbab-ca7c9c965b3f"). InnerVolumeSpecName "kube-api-access-6qlnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:45:03.321369 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.321340    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67d4700b-30b9-4570-bbab-ca7c9c965b3f" (UID: "67d4700b-30b9-4570-bbab-ca7c9c965b3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:45:03.406062 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.406035    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:45:03.406062 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.406060    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:45:03.406291 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.406070    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:45:03.406291 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.406080    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d4700b-30b9-4570-bbab-ca7c9c965b3f-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:45:03.406291 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.406089    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qlnk\" (UniqueName: \"kubernetes.io/projected/67d4700b-30b9-4570-bbab-ca7c9c965b3f-kube-api-access-6qlnk\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:45:03.537221 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.537163    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"]
Jul 27 18:45:03.541471 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:03.541451    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wcgmb"]
Jul 27 18:45:05.119969 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:45:05.119934    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" path="/var/lib/kubelet/pods/67d4700b-30b9-4570-bbab-ca7c9c965b3f/volumes"
Jul 27 18:46:54.446011 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.445912    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"]
Jul 27 18:46:54.446482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.446325    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy"
Jul 27 18:46:54.446482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.446339    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy"
Jul 27 18:46:54.446482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.446353    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="storage-initializer"
Jul 27 18:46:54.446482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.446359    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="storage-initializer"
Jul 27 18:46:54.446482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.446367    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kserve-container"
Jul 27 18:46:54.446482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.446372    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kserve-container"
Jul 27 18:46:54.446482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.446443    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kserve-container"
Jul 27 18:46:54.446482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.446453    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d4700b-30b9-4570-bbab-ca7c9c965b3f" containerName="kube-rbac-proxy"
Jul 27 18:46:54.449751 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.449732    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.453435 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.453404    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:46:54.453629 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.453587    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:46:54.454633 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.454615    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:46:54.454752 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.454727    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:46:54.454861 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.454781    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 27 18:46:54.454861 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.454807    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:46:54.454966 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.454789    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 27 18:46:54.461789 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.461760    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"]
Jul 27 18:46:54.566518 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.566467    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wmrn\" (UniqueName: \"kubernetes.io/projected/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-api-access-7wmrn\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.566518 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.566516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.566803 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.566669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.566803 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.566702    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.566803 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.566724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.668176 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.668126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.668176 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.668177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.668549 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.668237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.668549 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.668272    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wmrn\" (UniqueName: \"kubernetes.io/projected/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-api-access-7wmrn\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.668549 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:46:54.668378    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Jul 27 18:46:54.668549 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.668402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.668549 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:46:54.668467    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-proxy-tls podName:6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3 nodeName:}" failed. No retries permitted until 2026-07-27 18:46:55.168443227 +0000 UTC m=+4758.621064356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-ztjkt" (UID: "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3") : secret "isvc-xgboost-predictor-serving-cert" not found
Jul 27 18:46:54.668861 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.668811    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.669034 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.669012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.669106 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.669082    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:54.677886 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:54.677856    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wmrn\" (UniqueName: \"kubernetes.io/projected/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-api-access-7wmrn\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:55.174106 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:55.174061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:55.176623 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:55.176588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ztjkt\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:55.361502 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:55.361464    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:46:55.508399 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:55.508372    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"]
Jul 27 18:46:55.511272 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:46:55.511240    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f125e64_0fa7_4fbc_ad2b_2655eda1f1a3.slice/crio-606dabc1cddf7ac50ff27b44c64e37171e83217b154c7e2a6adc14ce81d57940 WatchSource:0}: Error finding container 606dabc1cddf7ac50ff27b44c64e37171e83217b154c7e2a6adc14ce81d57940: Status 404 returned error can't find the container with id 606dabc1cddf7ac50ff27b44c64e37171e83217b154c7e2a6adc14ce81d57940
Jul 27 18:46:55.687309 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:55.687272    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" event={"ID":"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3","Type":"ContainerStarted","Data":"84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e"}
Jul 27 18:46:55.687586 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:55.687335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" event={"ID":"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3","Type":"ContainerStarted","Data":"606dabc1cddf7ac50ff27b44c64e37171e83217b154c7e2a6adc14ce81d57940"}
Jul 27 18:46:59.705657 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:59.705556    2564 generic.go:356] "Generic (PLEG): container finished" podID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerID="84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e" exitCode=0
Jul 27 18:46:59.705657 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:46:59.705630    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" event={"ID":"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3","Type":"ContainerDied","Data":"84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e"}
Jul 27 18:47:16.798395 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:16.798355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" event={"ID":"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3","Type":"ContainerStarted","Data":"3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944"}
Jul 27 18:47:17.804827 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:17.804789    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" event={"ID":"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3","Type":"ContainerStarted","Data":"1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d"}
Jul 27 18:47:17.805302 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:17.805043    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:47:17.805302 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:17.805103    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:47:17.806586 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:17.806552    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 27 18:47:17.825780 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:17.825720    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podStartSLOduration=3.136170116 podStartE2EDuration="23.825700248s" totalImagesPullingTime="16.977831891s" totalInitContainerRuntime="3.711698241s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:47:17.824135224 +0000 UTC m=+4781.276756397" watchObservedRunningTime="2026-07-27 18:47:17.825700248 +0000 UTC m=+4781.278321395"
Jul 27 18:47:18.809681 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:18.809637    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 27 18:47:23.814523 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:23.814489    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:47:23.815175 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:23.815143    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 27 18:47:33.816078 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:33.816028    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 27 18:47:37.339009 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:37.338888    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:47:37.608072 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:37.346929    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:47:37.608072 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:37.349590    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:47:37.608072 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:37.356639    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:47:43.816005 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:43.815956    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 27 18:47:53.816088 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:47:53.816040    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 27 18:48:03.815938 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:03.815887    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 27 18:48:13.815803 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:13.815757    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 27 18:48:23.816145 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:23.816102    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:48:24.602485 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:24.602441    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"]
Jul 27 18:48:24.602725 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:24.602697    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container" containerID="cri-o://3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944" gracePeriod=30
Jul 27 18:48:24.602797 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:24.602740    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kube-rbac-proxy" containerID="cri-o://1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d" gracePeriod=30
Jul 27 18:48:25.092383 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:25.092346    2564 generic.go:356] "Generic (PLEG): container finished" podID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerID="1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d" exitCode=2
Jul 27 18:48:25.092825 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:25.092419    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" event={"ID":"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3","Type":"ContainerDied","Data":"1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d"}
Jul 27 18:48:28.854331 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.854300    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:48:28.944998 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.944959    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wmrn\" (UniqueName: \"kubernetes.io/projected/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-api-access-7wmrn\") pod \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") "
Jul 27 18:48:28.945230 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.945011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-cabundle-cert\") pod \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") "
Jul 27 18:48:28.945230 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.945058    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-proxy-tls\") pod \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") "
Jul 27 18:48:28.945230 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.945078    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kserve-provision-location\") pod \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") "
Jul 27 18:48:28.945230 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.945128    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-rbac-proxy-sar-config\") pod \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\" (UID: \"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3\") "
Jul 27 18:48:28.945529 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.945492    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" (UID: "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:48:28.945646 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.945546    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" (UID: "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:48:28.945695 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.945662    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" (UID: "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:48:28.947431 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.947377    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" (UID: "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:48:28.947431 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:28.947399    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-api-access-7wmrn" (OuterVolumeSpecName: "kube-api-access-7wmrn") pod "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" (UID: "6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3"). InnerVolumeSpecName "kube-api-access-7wmrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:48:29.046527 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.046479    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wmrn\" (UniqueName: \"kubernetes.io/projected/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-api-access-7wmrn\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:48:29.046527 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.046521    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:48:29.046527 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.046532    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:48:29.046809 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.046540    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:48:29.046809 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.046553    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:48:29.114200 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.114145    2564 generic.go:356] "Generic (PLEG): container finished" podID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerID="3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944" exitCode=0
Jul 27 18:48:29.114385 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.114253    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"
Jul 27 18:48:29.120059 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.120025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" event={"ID":"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3","Type":"ContainerDied","Data":"3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944"}
Jul 27 18:48:29.120294 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.120084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" event={"ID":"6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3","Type":"ContainerDied","Data":"606dabc1cddf7ac50ff27b44c64e37171e83217b154c7e2a6adc14ce81d57940"}
Jul 27 18:48:29.120294 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.120108    2564 scope.go:117] "RemoveContainer" containerID="1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d"
Jul 27 18:48:29.129709 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.129684    2564 scope.go:117] "RemoveContainer" containerID="3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944"
Jul 27 18:48:29.138147 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.138121    2564 scope.go:117] "RemoveContainer" containerID="84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e"
Jul 27 18:48:29.143010 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.142964    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"]
Jul 27 18:48:29.145567 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.145538    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt"]
Jul 27 18:48:29.150292 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.150269    2564 scope.go:117] "RemoveContainer" containerID="1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d"
Jul 27 18:48:29.150781 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:48:29.150754    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d\": container with ID starting with 1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d not found: ID does not exist" containerID="1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d"
Jul 27 18:48:29.150877 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.150795    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d"} err="failed to get container status \"1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d\": rpc error: code = NotFound desc = could not find container \"1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d\": container with ID starting with 1883829657d37e9d4893937046b6e612eedba70fbfd645505b608a225370fa9d not found: ID does not exist"
Jul 27 18:48:29.150877 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.150825    2564 scope.go:117] "RemoveContainer" containerID="3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944"
Jul 27 18:48:29.151201 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:48:29.151158    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944\": container with ID starting with 3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944 not found: ID does not exist" containerID="3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944"
Jul 27 18:48:29.151269 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.151198    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944"} err="failed to get container status \"3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944\": rpc error: code = NotFound desc = could not find container \"3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944\": container with ID starting with 3e4cb38455750d6662ad227320b16a87b2f02bf5bf4d8c09702ca6283c7ac944 not found: ID does not exist"
Jul 27 18:48:29.151269 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.151218    2564 scope.go:117] "RemoveContainer" containerID="84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e"
Jul 27 18:48:29.151522 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:48:29.151494    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e\": container with ID starting with 84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e not found: ID does not exist" containerID="84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e"
Jul 27 18:48:29.151584 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.151533    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e"} err="failed to get container status \"84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e\": rpc error: code = NotFound desc = could not find container \"84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e\": container with ID starting with 84c0ee6af23cbafc6f6f76e3b55dba970c40c452e64356b1fc790e32c1699d6e not found: ID does not exist"
Jul 27 18:48:29.811010 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:29.810946    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ztjkt" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 18:48:31.121292 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:48:31.121246    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" path="/var/lib/kubelet/pods/6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3/volumes"
Jul 27 18:50:20.214893 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.214789    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"]
Jul 27 18:50:20.215474 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.215287    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="storage-initializer"
Jul 27 18:50:20.215474 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.215308    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="storage-initializer"
Jul 27 18:50:20.215474 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.215336    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container"
Jul 27 18:50:20.215474 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.215345    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container"
Jul 27 18:50:20.215474 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.215374    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kube-rbac-proxy"
Jul 27 18:50:20.215474 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.215383    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kube-rbac-proxy"
Jul 27 18:50:20.215474 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.215463    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kube-rbac-proxy"
Jul 27 18:50:20.215474 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.215476    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f125e64-0fa7-4fbc-ad2b-2655eda1f1a3" containerName="kserve-container"
Jul 27 18:50:20.217924 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.217898    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.220773 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.220748    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 27 18:50:20.221108 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.221081    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:50:20.221219 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.221083    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:50:20.221219 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.221095    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:50:20.222080 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.222062    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:50:20.222208 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.222076    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:50:20.222208 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.222144    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:50:20.229178 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.229148    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"]
Jul 27 18:50:20.400587 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.400533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252e6750-9ee6-4d7e-a773-b308068a00c9-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.400791 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.400682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252e6750-9ee6-4d7e-a773-b308068a00c9-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.400791 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.400733    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.400874 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.400811    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.400874 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.400857    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5pd\" (UniqueName: \"kubernetes.io/projected/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-api-access-sl5pd\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.501873 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.501832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252e6750-9ee6-4d7e-a773-b308068a00c9-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.502068 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.501902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252e6750-9ee6-4d7e-a773-b308068a00c9-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.502068 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.501927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.502068 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.501961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.502068 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.502013    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5pd\" (UniqueName: \"kubernetes.io/projected/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-api-access-sl5pd\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.502517 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.502494    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252e6750-9ee6-4d7e-a773-b308068a00c9-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.502824 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.502800    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.502869 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.502848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.504541 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.504520    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252e6750-9ee6-4d7e-a773-b308068a00c9-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.510317 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.510287    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5pd\" (UniqueName: \"kubernetes.io/projected/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-api-access-sl5pd\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-q58bz\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.531298 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.531251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:20.679080 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:20.679042    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"]
Jul 27 18:50:20.682481 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:50:20.682450    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252e6750_9ee6_4d7e_a773_b308068a00c9.slice/crio-1a6a76730245b8eba2333913afb399445cce0f537270eb52f7590e63eb33729f WatchSource:0}: Error finding container 1a6a76730245b8eba2333913afb399445cce0f537270eb52f7590e63eb33729f: Status 404 returned error can't find the container with id 1a6a76730245b8eba2333913afb399445cce0f537270eb52f7590e63eb33729f
Jul 27 18:50:21.598959 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:21.598922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" event={"ID":"252e6750-9ee6-4d7e-a773-b308068a00c9","Type":"ContainerStarted","Data":"2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0"}
Jul 27 18:50:21.599431 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:21.598984    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" event={"ID":"252e6750-9ee6-4d7e-a773-b308068a00c9","Type":"ContainerStarted","Data":"1a6a76730245b8eba2333913afb399445cce0f537270eb52f7590e63eb33729f"}
Jul 27 18:50:24.613264 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:24.613139    2564 generic.go:356] "Generic (PLEG): container finished" podID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerID="2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0" exitCode=0
Jul 27 18:50:24.613264 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:24.613212    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" event={"ID":"252e6750-9ee6-4d7e-a773-b308068a00c9","Type":"ContainerDied","Data":"2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0"}
Jul 27 18:50:25.619800 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:25.619763    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" event={"ID":"252e6750-9ee6-4d7e-a773-b308068a00c9","Type":"ContainerStarted","Data":"0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b"}
Jul 27 18:50:25.620335 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:25.619833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" event={"ID":"252e6750-9ee6-4d7e-a773-b308068a00c9","Type":"ContainerStarted","Data":"9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e"}
Jul 27 18:50:25.620335 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:25.620051    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:25.640354 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:25.640296    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podStartSLOduration=2.096779282 podStartE2EDuration="5.640278593s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.543499311s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:50:25.639042061 +0000 UTC m=+4969.091663230" watchObservedRunningTime="2026-07-27 18:50:25.640278593 +0000 UTC m=+4969.092899738"
Jul 27 18:50:26.624729 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:26.624691    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:26.625994 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:26.625960    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 18:50:27.629011 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:27.628967    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 18:50:32.633799 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:32.633764    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:50:32.634307 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:32.634281    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 18:50:42.635218 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:42.635146    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 18:50:52.634707 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:50:52.634660    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 18:51:02.640039 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:02.639973    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 18:51:12.635166 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:12.635113    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 18:51:22.634955 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:22.634904    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 18:51:32.635338 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:32.635251    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:51:40.348668 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:40.348627    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"]
Jul 27 18:51:40.349369 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:40.348963    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" containerID="cri-o://9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e" gracePeriod=30
Jul 27 18:51:40.349369 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:40.349035    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kube-rbac-proxy" containerID="cri-o://0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b" gracePeriod=30
Jul 27 18:51:40.939229 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:40.939176    2564 generic.go:356] "Generic (PLEG): container finished" podID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerID="0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b" exitCode=2
Jul 27 18:51:40.939448 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:40.939253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" event={"ID":"252e6750-9ee6-4d7e-a773-b308068a00c9","Type":"ContainerDied","Data":"0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b"}
Jul 27 18:51:42.629915 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:42.629867    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 27 18:51:42.635368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:42.635325    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 27 18:51:44.809173 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.809141    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:51:44.959345 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.959249    2564 generic.go:356] "Generic (PLEG): container finished" podID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerID="9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e" exitCode=0
Jul 27 18:51:44.959345 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.959310    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"
Jul 27 18:51:44.959345 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.959335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" event={"ID":"252e6750-9ee6-4d7e-a773-b308068a00c9","Type":"ContainerDied","Data":"9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e"}
Jul 27 18:51:44.959589 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.959408    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz" event={"ID":"252e6750-9ee6-4d7e-a773-b308068a00c9","Type":"ContainerDied","Data":"1a6a76730245b8eba2333913afb399445cce0f537270eb52f7590e63eb33729f"}
Jul 27 18:51:44.959589 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.959438    2564 scope.go:117] "RemoveContainer" containerID="0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b"
Jul 27 18:51:44.965733 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.965704    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252e6750-9ee6-4d7e-a773-b308068a00c9-kserve-provision-location\") pod \"252e6750-9ee6-4d7e-a773-b308068a00c9\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") "
Jul 27 18:51:44.965883 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.965831    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252e6750-9ee6-4d7e-a773-b308068a00c9-proxy-tls\") pod \"252e6750-9ee6-4d7e-a773-b308068a00c9\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") "
Jul 27 18:51:44.965883 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.965865    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-rbac-proxy-sar-config\") pod \"252e6750-9ee6-4d7e-a773-b308068a00c9\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") "
Jul 27 18:51:44.965956 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.965895    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl5pd\" (UniqueName: \"kubernetes.io/projected/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-api-access-sl5pd\") pod \"252e6750-9ee6-4d7e-a773-b308068a00c9\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") "
Jul 27 18:51:44.965956 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.965939    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-cabundle-cert\") pod \"252e6750-9ee6-4d7e-a773-b308068a00c9\" (UID: \"252e6750-9ee6-4d7e-a773-b308068a00c9\") "
Jul 27 18:51:44.966081 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.966046    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252e6750-9ee6-4d7e-a773-b308068a00c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "252e6750-9ee6-4d7e-a773-b308068a00c9" (UID: "252e6750-9ee6-4d7e-a773-b308068a00c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:51:44.966307 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.966285    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252e6750-9ee6-4d7e-a773-b308068a00c9-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:51:44.966416 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.966277    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "252e6750-9ee6-4d7e-a773-b308068a00c9" (UID: "252e6750-9ee6-4d7e-a773-b308068a00c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:51:44.966465 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.966438    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "252e6750-9ee6-4d7e-a773-b308068a00c9" (UID: "252e6750-9ee6-4d7e-a773-b308068a00c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:51:44.968017 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.967992    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-api-access-sl5pd" (OuterVolumeSpecName: "kube-api-access-sl5pd") pod "252e6750-9ee6-4d7e-a773-b308068a00c9" (UID: "252e6750-9ee6-4d7e-a773-b308068a00c9"). InnerVolumeSpecName "kube-api-access-sl5pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:51:44.968474 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.968448    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252e6750-9ee6-4d7e-a773-b308068a00c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "252e6750-9ee6-4d7e-a773-b308068a00c9" (UID: "252e6750-9ee6-4d7e-a773-b308068a00c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:51:44.969145 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.969123    2564 scope.go:117] "RemoveContainer" containerID="9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e"
Jul 27 18:51:44.983975 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.983946    2564 scope.go:117] "RemoveContainer" containerID="2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0"
Jul 27 18:51:44.992375 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.992351    2564 scope.go:117] "RemoveContainer" containerID="0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b"
Jul 27 18:51:44.992759 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:51:44.992730    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b\": container with ID starting with 0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b not found: ID does not exist" containerID="0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b"
Jul 27 18:51:44.992811 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.992770    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b"} err="failed to get container status \"0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b\": rpc error: code = NotFound desc = could not find container \"0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b\": container with ID starting with 0d7e1e2b65f32ea4d12b591ecb7d223b0168548f028c901b72de9fd7f3388f1b not found: ID does not exist"
Jul 27 18:51:44.992811 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.992792    2564 scope.go:117] "RemoveContainer" containerID="9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e"
Jul 27 18:51:44.993074 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:51:44.993054    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e\": container with ID starting with 9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e not found: ID does not exist" containerID="9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e"
Jul 27 18:51:44.993125 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.993081    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e"} err="failed to get container status \"9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e\": rpc error: code = NotFound desc = could not find container \"9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e\": container with ID starting with 9f37978992e547b6b4a0a925992bf3e90093547a35ed9c8d85d79ee331980a2e not found: ID does not exist"
Jul 27 18:51:44.993125 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.993099    2564 scope.go:117] "RemoveContainer" containerID="2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0"
Jul 27 18:51:44.993376 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:51:44.993353    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0\": container with ID starting with 2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0 not found: ID does not exist" containerID="2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0"
Jul 27 18:51:44.993488 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:44.993379    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0"} err="failed to get container status \"2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0\": rpc error: code = NotFound desc = could not find container \"2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0\": container with ID starting with 2b3ba83c02c14a47bf73458bd501b9a2c280e9dd040725b4eab1745ca9dc31d0 not found: ID does not exist"
Jul 27 18:51:45.066925 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:45.066880    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252e6750-9ee6-4d7e-a773-b308068a00c9-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:51:45.066925 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:45.066918    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:51:45.066925 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:45.066931    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl5pd\" (UniqueName: \"kubernetes.io/projected/252e6750-9ee6-4d7e-a773-b308068a00c9-kube-api-access-sl5pd\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:51:45.067222 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:45.066940    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252e6750-9ee6-4d7e-a773-b308068a00c9-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:51:45.278735 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:45.278690    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"]
Jul 27 18:51:45.282377 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:45.282344    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-q58bz"]
Jul 27 18:51:47.120627 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:51:47.120595    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" path="/var/lib/kubelet/pods/252e6750-9ee6-4d7e-a773-b308068a00c9/volumes"
Jul 27 18:52:37.368053 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:37.367932    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:52:37.375925 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:37.375899    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:52:37.376722 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:37.376703    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:52:37.384076 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:37.384047    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:52:50.746448 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.746401    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"]
Jul 27 18:52:50.746914 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.746783    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kube-rbac-proxy"
Jul 27 18:52:50.746914 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.746798    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kube-rbac-proxy"
Jul 27 18:52:50.746914 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.746837    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container"
Jul 27 18:52:50.746914 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.746844    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container"
Jul 27 18:52:50.746914 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.746853    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="storage-initializer"
Jul 27 18:52:50.746914 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.746863    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="storage-initializer"
Jul 27 18:52:50.746914 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.746919    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kserve-container"
Jul 27 18:52:50.747211 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.746928    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="252e6750-9ee6-4d7e-a773-b308068a00c9" containerName="kube-rbac-proxy"
Jul 27 18:52:50.751641 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.751620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.754482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.754449    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:52:50.754642 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.754460    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:52:50.754642 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.754511    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:52:50.754642 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.754567    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:52:50.755584 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.755567    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 27 18:52:50.755685 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.755590    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:52:50.755685 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.755672    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 27 18:52:50.760294 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.760265    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"]
Jul 27 18:52:50.844397 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.844357    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.844397 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.844399    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.844649 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.844436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfsm\" (UniqueName: \"kubernetes.io/projected/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-api-access-trfsm\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.844649 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.844541    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.844649 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.844593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.945530 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.945487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfsm\" (UniqueName: \"kubernetes.io/projected/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-api-access-trfsm\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.945719 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.945556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.945719 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.945612    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.945719 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.945643    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.945863 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.945759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.946255 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.946226    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.946482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.946463    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.946535 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.946490    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.948142 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.948113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:50.953677 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:50.953651    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfsm\" (UniqueName: \"kubernetes.io/projected/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-api-access-trfsm\") pod \"isvc-xgboost-v2-predictor-fcc668bc-q9jl6\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:51.063551 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:51.063443    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:51.210319 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:51.210291    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"]
Jul 27 18:52:51.213052 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:52:51.213020    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa2ec07_36ba_44cf_ae39_b593d9c37bc2.slice/crio-105cf39efe1e56c1ea0fcb9e65e38acc44649c3d56e7471bf32feddae79648d8 WatchSource:0}: Error finding container 105cf39efe1e56c1ea0fcb9e65e38acc44649c3d56e7471bf32feddae79648d8: Status 404 returned error can't find the container with id 105cf39efe1e56c1ea0fcb9e65e38acc44649c3d56e7471bf32feddae79648d8
Jul 27 18:52:51.241880 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:51.241842    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" event={"ID":"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2","Type":"ContainerStarted","Data":"105cf39efe1e56c1ea0fcb9e65e38acc44649c3d56e7471bf32feddae79648d8"}
Jul 27 18:52:52.248037 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:52.247995    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" event={"ID":"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2","Type":"ContainerStarted","Data":"6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2"}
Jul 27 18:52:55.262054 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:55.262020    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerID="6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2" exitCode=0
Jul 27 18:52:55.262486 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:55.262061    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" event={"ID":"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2","Type":"ContainerDied","Data":"6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2"}
Jul 27 18:52:56.268395 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:56.268355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" event={"ID":"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2","Type":"ContainerStarted","Data":"519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4"}
Jul 27 18:52:56.268924 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:56.268449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" event={"ID":"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2","Type":"ContainerStarted","Data":"d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293"}
Jul 27 18:52:56.268924 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:56.268731    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:56.268924 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:56.268775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:52:56.270167 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:56.270134    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 18:52:56.289999 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:56.289943    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podStartSLOduration=2.4134448490000002 podStartE2EDuration="6.28992308s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.876478231s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:52:56.288266098 +0000 UTC m=+5119.740887244" watchObservedRunningTime="2026-07-27 18:52:56.28992308 +0000 UTC m=+5119.742544227"
Jul 27 18:52:57.274393 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:52:57.274346    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 18:53:02.278705 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:53:02.278622    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:53:02.279349 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:53:02.279320    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 18:53:12.279477 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:53:12.279431    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 18:53:22.279270 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:53:22.279223    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 18:53:32.279949 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:53:32.279902    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 18:53:42.279859 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:53:42.279799    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 18:53:52.279339 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:53:52.279288    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 18:54:02.280347 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:02.280314    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:54:10.874730 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:10.874687    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"]
Jul 27 18:54:10.875329 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:10.875024    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" containerID="cri-o://d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293" gracePeriod=30
Jul 27 18:54:10.875329 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:10.875095    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kube-rbac-proxy" containerID="cri-o://519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4" gracePeriod=30
Jul 27 18:54:11.597867 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:11.597821    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerID="519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4" exitCode=2
Jul 27 18:54:11.597867 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:11.597852    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" event={"ID":"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2","Type":"ContainerDied","Data":"519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4"}
Jul 27 18:54:12.274636 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:12.274586    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 27 18:54:12.279976 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:12.279939    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 27 18:54:15.439806 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.439775    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:54:15.529107 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529071    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trfsm\" (UniqueName: \"kubernetes.io/projected/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-api-access-trfsm\") pod \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") "
Jul 27 18:54:15.529342 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529131    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-rbac-proxy-sar-config\") pod \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") "
Jul 27 18:54:15.529342 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529159    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-proxy-tls\") pod \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") "
Jul 27 18:54:15.529342 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529217    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-cabundle-cert\") pod \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") "
Jul 27 18:54:15.529519 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529351    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kserve-provision-location\") pod \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\" (UID: \"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2\") "
Jul 27 18:54:15.529609 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529580    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" (UID: "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:54:15.529656 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529589    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" (UID: "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:54:15.529703 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529656    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" (UID: "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:54:15.529750 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529701    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:54:15.529750 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.529717    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:54:15.531429 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.531401    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" (UID: "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:54:15.531545 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.531523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-api-access-trfsm" (OuterVolumeSpecName: "kube-api-access-trfsm") pod "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" (UID: "0fa2ec07-36ba-44cf-ae39-b593d9c37bc2"). InnerVolumeSpecName "kube-api-access-trfsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:54:15.618886 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.618844    2564 generic.go:356] "Generic (PLEG): container finished" podID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerID="d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293" exitCode=0
Jul 27 18:54:15.619085 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.618907    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"
Jul 27 18:54:15.619085 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.618915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" event={"ID":"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2","Type":"ContainerDied","Data":"d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293"}
Jul 27 18:54:15.619085 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.618985    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6" event={"ID":"0fa2ec07-36ba-44cf-ae39-b593d9c37bc2","Type":"ContainerDied","Data":"105cf39efe1e56c1ea0fcb9e65e38acc44649c3d56e7471bf32feddae79648d8"}
Jul 27 18:54:15.619085 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.619013    2564 scope.go:117] "RemoveContainer" containerID="519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4"
Jul 27 18:54:15.628260 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.628240    2564 scope.go:117] "RemoveContainer" containerID="d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293"
Jul 27 18:54:15.630702 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.630682    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trfsm\" (UniqueName: \"kubernetes.io/projected/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kube-api-access-trfsm\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:54:15.630826 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.630705    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:54:15.630826 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.630717    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:54:15.636845 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.636822    2564 scope.go:117] "RemoveContainer" containerID="6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2"
Jul 27 18:54:15.641960 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.641930    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"]
Jul 27 18:54:15.646267 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.646240    2564 scope.go:117] "RemoveContainer" containerID="519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4"
Jul 27 18:54:15.646637 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:54:15.646614    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4\": container with ID starting with 519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4 not found: ID does not exist" containerID="519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4"
Jul 27 18:54:15.646703 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.646650    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4"} err="failed to get container status \"519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4\": rpc error: code = NotFound desc = could not find container \"519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4\": container with ID starting with 519cb98c2481e5a3496d51ced97fd4d628304e2f52ea9368de984ae093a998f4 not found: ID does not exist"
Jul 27 18:54:15.646703 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.646663    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-q9jl6"]
Jul 27 18:54:15.646703 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.646676    2564 scope.go:117] "RemoveContainer" containerID="d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293"
Jul 27 18:54:15.646946 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:54:15.646926    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293\": container with ID starting with d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293 not found: ID does not exist" containerID="d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293"
Jul 27 18:54:15.647014 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.646956    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293"} err="failed to get container status \"d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293\": rpc error: code = NotFound desc = could not find container \"d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293\": container with ID starting with d6a221ec9ff52b4f77653af6682032d726e0ddc2eb15a5a75f458ccc74c98293 not found: ID does not exist"
Jul 27 18:54:15.647014 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.646980    2564 scope.go:117] "RemoveContainer" containerID="6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2"
Jul 27 18:54:15.647297 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:54:15.647278    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2\": container with ID starting with 6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2 not found: ID does not exist" containerID="6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2"
Jul 27 18:54:15.647368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:15.647300    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2"} err="failed to get container status \"6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2\": rpc error: code = NotFound desc = could not find container \"6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2\": container with ID starting with 6f0b14b7cec92f0946a3a440df60276e27521ffcbc566e6a016e73e808b640a2 not found: ID does not exist"
Jul 27 18:54:16.020376 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.020336    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"]
Jul 27 18:54:16.020687 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.020673    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="storage-initializer"
Jul 27 18:54:16.020743 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.020688    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="storage-initializer"
Jul 27 18:54:16.020743 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.020698    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kube-rbac-proxy"
Jul 27 18:54:16.020743 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.020704    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kube-rbac-proxy"
Jul 27 18:54:16.020743 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.020725    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container"
Jul 27 18:54:16.020743 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.020731    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container"
Jul 27 18:54:16.020924 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.020780    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kserve-container"
Jul 27 18:54:16.020924 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.020789    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" containerName="kube-rbac-proxy"
Jul 27 18:54:16.025376 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.025350    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.028238 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.028208    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 27 18:54:16.028405 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.028208    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 27 18:54:16.028405 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.028352    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 27 18:54:16.028405 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.028364    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:54:16.028405 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.028380    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:54:16.029208 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.029194    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:54:16.033794 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.033765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"]
Jul 27 18:54:16.034675 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.034648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0571bae-0f15-49a3-90e7-e23859a60d6f-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.034802 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.034767    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbzb5\" (UniqueName: \"kubernetes.io/projected/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-api-access-lbzb5\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.034866 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.034823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0571bae-0f15-49a3-90e7-e23859a60d6f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.034930 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.034892    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.136055 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.136004    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbzb5\" (UniqueName: \"kubernetes.io/projected/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-api-access-lbzb5\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.136055 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.136058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0571bae-0f15-49a3-90e7-e23859a60d6f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.136394 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.136262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.136394 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.136312    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0571bae-0f15-49a3-90e7-e23859a60d6f-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.136791 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.136770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0571bae-0f15-49a3-90e7-e23859a60d6f-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.136977 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.136957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.138695 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.138677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0571bae-0f15-49a3-90e7-e23859a60d6f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.144997 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.144971    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbzb5\" (UniqueName: \"kubernetes.io/projected/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-api-access-lbzb5\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-npg2r\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.338388 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.338267    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:16.479972 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.479895    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"]
Jul 27 18:54:16.482905 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:54:16.482870    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0571bae_0f15_49a3_90e7_e23859a60d6f.slice/crio-a2bfadaa87a054434b575c55658667812ddc5a70d3377f377eec17459e4d7224 WatchSource:0}: Error finding container a2bfadaa87a054434b575c55658667812ddc5a70d3377f377eec17459e4d7224: Status 404 returned error can't find the container with id a2bfadaa87a054434b575c55658667812ddc5a70d3377f377eec17459e4d7224
Jul 27 18:54:16.628029 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.627927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" event={"ID":"d0571bae-0f15-49a3-90e7-e23859a60d6f","Type":"ContainerStarted","Data":"6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746"}
Jul 27 18:54:16.628029 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:16.627992    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" event={"ID":"d0571bae-0f15-49a3-90e7-e23859a60d6f","Type":"ContainerStarted","Data":"a2bfadaa87a054434b575c55658667812ddc5a70d3377f377eec17459e4d7224"}
Jul 27 18:54:17.120098 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:17.120056    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fa2ec07-36ba-44cf-ae39-b593d9c37bc2" path="/var/lib/kubelet/pods/0fa2ec07-36ba-44cf-ae39-b593d9c37bc2/volumes"
Jul 27 18:54:17.634491 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:17.634452    2564 generic.go:356] "Generic (PLEG): container finished" podID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerID="6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746" exitCode=0
Jul 27 18:54:17.634958 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:17.634524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" event={"ID":"d0571bae-0f15-49a3-90e7-e23859a60d6f","Type":"ContainerDied","Data":"6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746"}
Jul 27 18:54:18.641457 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:18.641420    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" event={"ID":"d0571bae-0f15-49a3-90e7-e23859a60d6f","Type":"ContainerStarted","Data":"69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8"}
Jul 27 18:54:18.641975 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:18.641495    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" event={"ID":"d0571bae-0f15-49a3-90e7-e23859a60d6f","Type":"ContainerStarted","Data":"3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7"}
Jul 27 18:54:18.641975 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:18.641630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:18.661419 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:18.661360    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podStartSLOduration=1.612593881 podStartE2EDuration="2.66134034s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.048746459s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:54:18.659803539 +0000 UTC m=+5202.112424684" watchObservedRunningTime="2026-07-27 18:54:18.66134034 +0000 UTC m=+5202.113961486"
Jul 27 18:54:19.646142 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:19.646099    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:19.647514 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:19.647486    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:54:20.651386 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:20.651342    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:54:25.656396 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:25.656364    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:54:25.657005 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:25.656974    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:54:35.657459 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:35.657356    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:54:45.657500 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:45.657450    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:54:55.657007 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:54:55.656962    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:55:05.657743 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:05.657692    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:55:15.656988 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:15.656938    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:55:25.657077 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:25.657033    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:55:28.115436 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:28.115394    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:55:38.116252 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:38.116217    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:55:46.160043 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.159995    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"]
Jul 27 18:55:46.160581 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.160276    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" containerID="cri-o://3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7" gracePeriod=30
Jul 27 18:55:46.160581 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.160335    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kube-rbac-proxy" containerID="cri-o://69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8" gracePeriod=30
Jul 27 18:55:46.388586 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.388546    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"]
Jul 27 18:55:46.392071 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.392049    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.395125 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.395093    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:55:46.395288 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.395098    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 27 18:55:46.395288 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.395199    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 27 18:55:46.403987 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.403951    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"]
Jul 27 18:55:46.459218 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.459075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.459405 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.459224    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.459405 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.459264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbn65\" (UniqueName: \"kubernetes.io/projected/a366b364-f00d-4661-84ac-fc800a48ad58-kube-api-access-bbn65\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.459405 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.459280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a366b364-f00d-4661-84ac-fc800a48ad58-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.459405 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.459349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a366b364-f00d-4661-84ac-fc800a48ad58-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.560719 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.560673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.560719 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.560726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbn65\" (UniqueName: \"kubernetes.io/projected/a366b364-f00d-4661-84ac-fc800a48ad58-kube-api-access-bbn65\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.561011 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.560753    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a366b364-f00d-4661-84ac-fc800a48ad58-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.561011 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.560796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a366b364-f00d-4661-84ac-fc800a48ad58-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.561011 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.560852    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.561011 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:55:46.560931    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 27 18:55:46.561337 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:55:46.561020    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a366b364-f00d-4661-84ac-fc800a48ad58-proxy-tls podName:a366b364-f00d-4661-84ac-fc800a48ad58 nodeName:}" failed. No retries permitted until 2026-07-27 18:55:47.060997788 +0000 UTC m=+5290.513618929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a366b364-f00d-4661-84ac-fc800a48ad58-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" (UID: "a366b364-f00d-4661-84ac-fc800a48ad58") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 27 18:55:46.561337 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.561268    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a366b364-f00d-4661-84ac-fc800a48ad58-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.561564 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.561542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.561690 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.561671    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:46.572243 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:46.572209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbn65\" (UniqueName: \"kubernetes.io/projected/a366b364-f00d-4661-84ac-fc800a48ad58-kube-api-access-bbn65\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:47.026616 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:47.026579    2564 generic.go:356] "Generic (PLEG): container finished" podID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerID="69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8" exitCode=2
Jul 27 18:55:47.026815 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:47.026652    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" event={"ID":"d0571bae-0f15-49a3-90e7-e23859a60d6f","Type":"ContainerDied","Data":"69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8"}
Jul 27 18:55:47.065249 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:47.065178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a366b364-f00d-4661-84ac-fc800a48ad58-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:47.067854 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:47.067829    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a366b364-f00d-4661-84ac-fc800a48ad58-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:47.304899 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:47.304791    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:47.461921 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:47.461894    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"]
Jul 27 18:55:47.464729 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:55:47.464693    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda366b364_f00d_4661_84ac_fc800a48ad58.slice/crio-6642a7d69a3e43e83cd52196319cfa6f07eaa80bceb19f33cee5207fe38956e8 WatchSource:0}: Error finding container 6642a7d69a3e43e83cd52196319cfa6f07eaa80bceb19f33cee5207fe38956e8: Status 404 returned error can't find the container with id 6642a7d69a3e43e83cd52196319cfa6f07eaa80bceb19f33cee5207fe38956e8
Jul 27 18:55:48.033452 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:48.033411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" event={"ID":"a366b364-f00d-4661-84ac-fc800a48ad58","Type":"ContainerStarted","Data":"ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7"}
Jul 27 18:55:48.033700 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:48.033477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" event={"ID":"a366b364-f00d-4661-84ac-fc800a48ad58","Type":"ContainerStarted","Data":"6642a7d69a3e43e83cd52196319cfa6f07eaa80bceb19f33cee5207fe38956e8"}
Jul 27 18:55:48.115919 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:48.115857    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 27 18:55:49.039692 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:49.039656    2564 generic.go:356] "Generic (PLEG): container finished" podID="a366b364-f00d-4661-84ac-fc800a48ad58" containerID="ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7" exitCode=0
Jul 27 18:55:49.040257 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:49.039745    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" event={"ID":"a366b364-f00d-4661-84ac-fc800a48ad58","Type":"ContainerDied","Data":"ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7"}
Jul 27 18:55:50.046523 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:50.046482    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" event={"ID":"a366b364-f00d-4661-84ac-fc800a48ad58","Type":"ContainerStarted","Data":"71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b"}
Jul 27 18:55:50.047007 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:50.046552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" event={"ID":"a366b364-f00d-4661-84ac-fc800a48ad58","Type":"ContainerStarted","Data":"492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186"}
Jul 27 18:55:50.047007 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:50.046662    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:50.047007 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:50.046725    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:50.048089 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:50.048062    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:55:50.068738 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:50.068679    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podStartSLOduration=2.998308465 podStartE2EDuration="4.068661872s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.070353407s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:55:50.067820334 +0000 UTC m=+5293.520441480" watchObservedRunningTime="2026-07-27 18:55:50.068661872 +0000 UTC m=+5293.521283017"
Jul 27 18:55:50.651725 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:50.651653    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 27 18:55:51.051161 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:51.051112    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:55:52.008288 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.008259    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:55:52.057330 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.057229    2564 generic.go:356] "Generic (PLEG): container finished" podID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerID="3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7" exitCode=0
Jul 27 18:55:52.057330 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.057292    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"
Jul 27 18:55:52.057330 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.057308    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" event={"ID":"d0571bae-0f15-49a3-90e7-e23859a60d6f","Type":"ContainerDied","Data":"3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7"}
Jul 27 18:55:52.057909 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.057377    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r" event={"ID":"d0571bae-0f15-49a3-90e7-e23859a60d6f","Type":"ContainerDied","Data":"a2bfadaa87a054434b575c55658667812ddc5a70d3377f377eec17459e4d7224"}
Jul 27 18:55:52.057909 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.057406    2564 scope.go:117] "RemoveContainer" containerID="69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8"
Jul 27 18:55:52.066074 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.066052    2564 scope.go:117] "RemoveContainer" containerID="3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7"
Jul 27 18:55:52.074212 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.074154    2564 scope.go:117] "RemoveContainer" containerID="6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746"
Jul 27 18:55:52.082283 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.082262    2564 scope.go:117] "RemoveContainer" containerID="69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8"
Jul 27 18:55:52.082578 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:55:52.082556    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8\": container with ID starting with 69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8 not found: ID does not exist" containerID="69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8"
Jul 27 18:55:52.082642 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.082592    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8"} err="failed to get container status \"69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8\": rpc error: code = NotFound desc = could not find container \"69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8\": container with ID starting with 69cc1d60eb206fa3264331a8f3aa3c988ac190166842dce5bb305a2a76ce45a8 not found: ID does not exist"
Jul 27 18:55:52.082642 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.082612    2564 scope.go:117] "RemoveContainer" containerID="3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7"
Jul 27 18:55:52.082856 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:55:52.082837    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7\": container with ID starting with 3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7 not found: ID does not exist" containerID="3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7"
Jul 27 18:55:52.082922 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.082866    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7"} err="failed to get container status \"3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7\": rpc error: code = NotFound desc = could not find container \"3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7\": container with ID starting with 3254cccc95a7041b558befcf9b783365216a3006576757b86d1c17a7716ffdd7 not found: ID does not exist"
Jul 27 18:55:52.082922 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.082889    2564 scope.go:117] "RemoveContainer" containerID="6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746"
Jul 27 18:55:52.083128 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:55:52.083112    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746\": container with ID starting with 6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746 not found: ID does not exist" containerID="6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746"
Jul 27 18:55:52.083204 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.083134    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746"} err="failed to get container status \"6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746\": rpc error: code = NotFound desc = could not find container \"6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746\": container with ID starting with 6a733ce80007e5575d8b2172036993ff32748db86eab4131f485b859a91d6746 not found: ID does not exist"
Jul 27 18:55:52.105597 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.105552    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0571bae-0f15-49a3-90e7-e23859a60d6f-proxy-tls\") pod \"d0571bae-0f15-49a3-90e7-e23859a60d6f\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") "
Jul 27 18:55:52.105771 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.105642    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbzb5\" (UniqueName: \"kubernetes.io/projected/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-api-access-lbzb5\") pod \"d0571bae-0f15-49a3-90e7-e23859a60d6f\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") "
Jul 27 18:55:52.105771 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.105678    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-rbac-proxy-sar-config\") pod \"d0571bae-0f15-49a3-90e7-e23859a60d6f\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") "
Jul 27 18:55:52.105771 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.105720    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0571bae-0f15-49a3-90e7-e23859a60d6f-kserve-provision-location\") pod \"d0571bae-0f15-49a3-90e7-e23859a60d6f\" (UID: \"d0571bae-0f15-49a3-90e7-e23859a60d6f\") "
Jul 27 18:55:52.106049 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.106022    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0571bae-0f15-49a3-90e7-e23859a60d6f" (UID: "d0571bae-0f15-49a3-90e7-e23859a60d6f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:55:52.106114 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.106051    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0571bae-0f15-49a3-90e7-e23859a60d6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0571bae-0f15-49a3-90e7-e23859a60d6f" (UID: "d0571bae-0f15-49a3-90e7-e23859a60d6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:55:52.107819 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.107798    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-api-access-lbzb5" (OuterVolumeSpecName: "kube-api-access-lbzb5") pod "d0571bae-0f15-49a3-90e7-e23859a60d6f" (UID: "d0571bae-0f15-49a3-90e7-e23859a60d6f"). InnerVolumeSpecName "kube-api-access-lbzb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:55:52.107819 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.107810    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0571bae-0f15-49a3-90e7-e23859a60d6f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0571bae-0f15-49a3-90e7-e23859a60d6f" (UID: "d0571bae-0f15-49a3-90e7-e23859a60d6f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:55:52.206400 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.206358    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0571bae-0f15-49a3-90e7-e23859a60d6f-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:55:52.206400 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.206391    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0571bae-0f15-49a3-90e7-e23859a60d6f-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:55:52.206400 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.206401    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbzb5\" (UniqueName: \"kubernetes.io/projected/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-api-access-lbzb5\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:55:52.206400 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.206410    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0571bae-0f15-49a3-90e7-e23859a60d6f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:55:52.380837 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.380796    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"]
Jul 27 18:55:52.384357 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:52.384328    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-npg2r"]
Jul 27 18:55:53.119940 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:53.119897    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" path="/var/lib/kubelet/pods/d0571bae-0f15-49a3-90e7-e23859a60d6f/volumes"
Jul 27 18:55:56.055396 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:56.055360    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:55:56.055895 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:55:56.055849    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:56:06.056671 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:56:06.056572    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:56:16.056324 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:56:16.056280    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:56:26.056013 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:56:26.055968    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:56:36.057121 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:56:36.057062    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:56:46.055978 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:56:46.055925    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:56:56.056766 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:56:56.056718    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:57:04.114961 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:04.114904    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 27 18:57:14.115840 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:14.115796    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:57:16.323453 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:16.323413    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"]
Jul 27 18:57:16.323965 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:16.323677    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container" containerID="cri-o://492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186" gracePeriod=30
Jul 27 18:57:16.323965 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:16.323739    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kube-rbac-proxy" containerID="cri-o://71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b" gracePeriod=30
Jul 27 18:57:16.443502 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:16.443468    2564 generic.go:356] "Generic (PLEG): container finished" podID="a366b364-f00d-4661-84ac-fc800a48ad58" containerID="71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b" exitCode=2
Jul 27 18:57:16.443635 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:16.443516    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" event={"ID":"a366b364-f00d-4661-84ac-fc800a48ad58","Type":"ContainerDied","Data":"71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b"}
Jul 27 18:57:17.447469 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.447434    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"]
Jul 27 18:57:17.448076 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.447785    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container"
Jul 27 18:57:17.448076 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.447802    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container"
Jul 27 18:57:17.448076 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.447829    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kube-rbac-proxy"
Jul 27 18:57:17.448076 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.447839    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kube-rbac-proxy"
Jul 27 18:57:17.448076 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.447856    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="storage-initializer"
Jul 27 18:57:17.448076 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.447865    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="storage-initializer"
Jul 27 18:57:17.448076 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.447945    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kserve-container"
Jul 27 18:57:17.448076 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.447957    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0571bae-0f15-49a3-90e7-e23859a60d6f" containerName="kube-rbac-proxy"
Jul 27 18:57:17.451156 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.451137    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.454018 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.453990    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 27 18:57:17.454137 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.454064    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 27 18:57:17.461953 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.461924    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"]
Jul 27 18:57:17.486152 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.486113    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/881385bf-de76-49c5-ad96-d01169bad70b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.486401 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.486213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881385bf-de76-49c5-ad96-d01169bad70b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.486401 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.486237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m5v7\" (UniqueName: \"kubernetes.io/projected/881385bf-de76-49c5-ad96-d01169bad70b-kube-api-access-5m5v7\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.486401 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.486264    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/881385bf-de76-49c5-ad96-d01169bad70b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.587549 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.587509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881385bf-de76-49c5-ad96-d01169bad70b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.587549 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.587554    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m5v7\" (UniqueName: \"kubernetes.io/projected/881385bf-de76-49c5-ad96-d01169bad70b-kube-api-access-5m5v7\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.587855 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.587582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/881385bf-de76-49c5-ad96-d01169bad70b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.587855 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.587626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/881385bf-de76-49c5-ad96-d01169bad70b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.587855 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:57:17.587698    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 27 18:57:17.587855 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:57:17.587790    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881385bf-de76-49c5-ad96-d01169bad70b-proxy-tls podName:881385bf-de76-49c5-ad96-d01169bad70b nodeName:}" failed. No retries permitted until 2026-07-27 18:57:18.087766152 +0000 UTC m=+5381.540387290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/881385bf-de76-49c5-ad96-d01169bad70b-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9" (UID: "881385bf-de76-49c5-ad96-d01169bad70b") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 27 18:57:17.588081 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.588058    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/881385bf-de76-49c5-ad96-d01169bad70b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.588416 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.588395    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/881385bf-de76-49c5-ad96-d01169bad70b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:17.598904 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:17.598870    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m5v7\" (UniqueName: \"kubernetes.io/projected/881385bf-de76-49c5-ad96-d01169bad70b-kube-api-access-5m5v7\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:18.093167 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:18.093118    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881385bf-de76-49c5-ad96-d01169bad70b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:18.095716 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:18.095684    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881385bf-de76-49c5-ad96-d01169bad70b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:18.362898 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:18.362790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:18.513223 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:18.513107    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"]
Jul 27 18:57:18.516383 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:57:18.516343    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881385bf_de76_49c5_ad96_d01169bad70b.slice/crio-c46afb8d6a9039649a77fdf55ce94bee29962577f24178c539c509e3daff0799 WatchSource:0}: Error finding container c46afb8d6a9039649a77fdf55ce94bee29962577f24178c539c509e3daff0799: Status 404 returned error can't find the container with id c46afb8d6a9039649a77fdf55ce94bee29962577f24178c539c509e3daff0799
Jul 27 18:57:19.459292 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:19.459252    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9" event={"ID":"881385bf-de76-49c5-ad96-d01169bad70b","Type":"ContainerStarted","Data":"2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc"}
Jul 27 18:57:19.459531 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:19.459315    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9" event={"ID":"881385bf-de76-49c5-ad96-d01169bad70b","Type":"ContainerStarted","Data":"c46afb8d6a9039649a77fdf55ce94bee29962577f24178c539c509e3daff0799"}
Jul 27 18:57:21.052229 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.052155    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 27 18:57:21.878272 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.878239    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:57:21.925649 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.925603    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-cabundle-cert\") pod \"a366b364-f00d-4661-84ac-fc800a48ad58\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") "
Jul 27 18:57:21.925649 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.925655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-kube-rbac-proxy-sar-config\") pod \"a366b364-f00d-4661-84ac-fc800a48ad58\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") "
Jul 27 18:57:21.925937 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.925709    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a366b364-f00d-4661-84ac-fc800a48ad58-kserve-provision-location\") pod \"a366b364-f00d-4661-84ac-fc800a48ad58\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") "
Jul 27 18:57:21.925937 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.925778    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbn65\" (UniqueName: \"kubernetes.io/projected/a366b364-f00d-4661-84ac-fc800a48ad58-kube-api-access-bbn65\") pod \"a366b364-f00d-4661-84ac-fc800a48ad58\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") "
Jul 27 18:57:21.925937 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.925807    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a366b364-f00d-4661-84ac-fc800a48ad58-proxy-tls\") pod \"a366b364-f00d-4661-84ac-fc800a48ad58\" (UID: \"a366b364-f00d-4661-84ac-fc800a48ad58\") "
Jul 27 18:57:21.926105 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.926075    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a366b364-f00d-4661-84ac-fc800a48ad58" (UID: "a366b364-f00d-4661-84ac-fc800a48ad58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:57:21.926153 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.926082    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a366b364-f00d-4661-84ac-fc800a48ad58" (UID: "a366b364-f00d-4661-84ac-fc800a48ad58"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:57:21.926153 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.926084    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a366b364-f00d-4661-84ac-fc800a48ad58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a366b364-f00d-4661-84ac-fc800a48ad58" (UID: "a366b364-f00d-4661-84ac-fc800a48ad58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:21.928011 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.927984    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a366b364-f00d-4661-84ac-fc800a48ad58-kube-api-access-bbn65" (OuterVolumeSpecName: "kube-api-access-bbn65") pod "a366b364-f00d-4661-84ac-fc800a48ad58" (UID: "a366b364-f00d-4661-84ac-fc800a48ad58"). InnerVolumeSpecName "kube-api-access-bbn65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:57:21.928126 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:21.928014    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a366b364-f00d-4661-84ac-fc800a48ad58-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a366b364-f00d-4661-84ac-fc800a48ad58" (UID: "a366b364-f00d-4661-84ac-fc800a48ad58"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:57:22.027583 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.027540    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:57:22.027583 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.027578    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a366b364-f00d-4661-84ac-fc800a48ad58-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:57:22.027813 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.027594    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a366b364-f00d-4661-84ac-fc800a48ad58-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:57:22.027813 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.027606    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbn65\" (UniqueName: \"kubernetes.io/projected/a366b364-f00d-4661-84ac-fc800a48ad58-kube-api-access-bbn65\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:57:22.027813 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.027616    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a366b364-f00d-4661-84ac-fc800a48ad58-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:57:22.475213 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.475154    2564 generic.go:356] "Generic (PLEG): container finished" podID="a366b364-f00d-4661-84ac-fc800a48ad58" containerID="492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186" exitCode=0
Jul 27 18:57:22.475678 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.475246    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"
Jul 27 18:57:22.475678 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.475238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" event={"ID":"a366b364-f00d-4661-84ac-fc800a48ad58","Type":"ContainerDied","Data":"492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186"}
Jul 27 18:57:22.475678 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.475406    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6" event={"ID":"a366b364-f00d-4661-84ac-fc800a48ad58","Type":"ContainerDied","Data":"6642a7d69a3e43e83cd52196319cfa6f07eaa80bceb19f33cee5207fe38956e8"}
Jul 27 18:57:22.475678 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.475442    2564 scope.go:117] "RemoveContainer" containerID="71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b"
Jul 27 18:57:22.477073 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.477049    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9_881385bf-de76-49c5-ad96-d01169bad70b/storage-initializer/0.log"
Jul 27 18:57:22.477220 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.477096    2564 generic.go:356] "Generic (PLEG): container finished" podID="881385bf-de76-49c5-ad96-d01169bad70b" containerID="2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc" exitCode=1
Jul 27 18:57:22.477220 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.477141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9" event={"ID":"881385bf-de76-49c5-ad96-d01169bad70b","Type":"ContainerDied","Data":"2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc"}
Jul 27 18:57:22.484774 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.484753    2564 scope.go:117] "RemoveContainer" containerID="492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186"
Jul 27 18:57:22.493427 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.493405    2564 scope.go:117] "RemoveContainer" containerID="ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7"
Jul 27 18:57:22.502862 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.502840    2564 scope.go:117] "RemoveContainer" containerID="71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b"
Jul 27 18:57:22.503210 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:57:22.503171    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b\": container with ID starting with 71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b not found: ID does not exist" containerID="71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b"
Jul 27 18:57:22.503276 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.503222    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b"} err="failed to get container status \"71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b\": rpc error: code = NotFound desc = could not find container \"71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b\": container with ID starting with 71b5a40d4a7871fa2b4497a2f7193da7c97f7a94629913f2a3e98fcaf60bb52b not found: ID does not exist"
Jul 27 18:57:22.503276 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.503243    2564 scope.go:117] "RemoveContainer" containerID="492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186"
Jul 27 18:57:22.503525 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:57:22.503508    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186\": container with ID starting with 492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186 not found: ID does not exist" containerID="492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186"
Jul 27 18:57:22.503568 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.503530    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186"} err="failed to get container status \"492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186\": rpc error: code = NotFound desc = could not find container \"492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186\": container with ID starting with 492c5dcc5dc3c93b229cc9457c585254a8e14016e6c64b5f3d1fc337daa7d186 not found: ID does not exist"
Jul 27 18:57:22.503568 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.503544    2564 scope.go:117] "RemoveContainer" containerID="ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7"
Jul 27 18:57:22.503867 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:57:22.503842    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7\": container with ID starting with ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7 not found: ID does not exist" containerID="ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7"
Jul 27 18:57:22.503939 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.503871    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7"} err="failed to get container status \"ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7\": rpc error: code = NotFound desc = could not find container \"ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7\": container with ID starting with ca0903fcc809127141810b9e53d4b954c0f6b4fa94b36e63411a20e126cadeb7 not found: ID does not exist"
Jul 27 18:57:22.516718 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.516598    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"]
Jul 27 18:57:22.520391 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:22.520359    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-mcfj6"]
Jul 27 18:57:23.120153 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:23.120113    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" path="/var/lib/kubelet/pods/a366b364-f00d-4661-84ac-fc800a48ad58/volumes"
Jul 27 18:57:23.484960 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:23.484925    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9_881385bf-de76-49c5-ad96-d01169bad70b/storage-initializer/0.log"
Jul 27 18:57:23.485439 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:23.485046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9" event={"ID":"881385bf-de76-49c5-ad96-d01169bad70b","Type":"ContainerStarted","Data":"4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f"}
Jul 27 18:57:27.408498 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:27.408451    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"]
Jul 27 18:57:27.408942 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:27.408726    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9" podUID="881385bf-de76-49c5-ad96-d01169bad70b" containerName="storage-initializer" containerID="cri-o://4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f" gracePeriod=30
Jul 27 18:57:27.964730 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:27.964706    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9_881385bf-de76-49c5-ad96-d01169bad70b/storage-initializer/1.log"
Jul 27 18:57:27.965085 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:27.965068    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9_881385bf-de76-49c5-ad96-d01169bad70b/storage-initializer/0.log"
Jul 27 18:57:27.967482 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:27.967463    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:28.085117 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.085011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881385bf-de76-49c5-ad96-d01169bad70b-proxy-tls\") pod \"881385bf-de76-49c5-ad96-d01169bad70b\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") "
Jul 27 18:57:28.085117 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.085082    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/881385bf-de76-49c5-ad96-d01169bad70b-kserve-provision-location\") pod \"881385bf-de76-49c5-ad96-d01169bad70b\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") "
Jul 27 18:57:28.085117 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.085116    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m5v7\" (UniqueName: \"kubernetes.io/projected/881385bf-de76-49c5-ad96-d01169bad70b-kube-api-access-5m5v7\") pod \"881385bf-de76-49c5-ad96-d01169bad70b\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") "
Jul 27 18:57:28.085462 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.085133    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/881385bf-de76-49c5-ad96-d01169bad70b-kube-rbac-proxy-sar-config\") pod \"881385bf-de76-49c5-ad96-d01169bad70b\" (UID: \"881385bf-de76-49c5-ad96-d01169bad70b\") "
Jul 27 18:57:28.085462 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.085426    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881385bf-de76-49c5-ad96-d01169bad70b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "881385bf-de76-49c5-ad96-d01169bad70b" (UID: "881385bf-de76-49c5-ad96-d01169bad70b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:28.085643 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.085614    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881385bf-de76-49c5-ad96-d01169bad70b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "881385bf-de76-49c5-ad96-d01169bad70b" (UID: "881385bf-de76-49c5-ad96-d01169bad70b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:57:28.087339 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.087312    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881385bf-de76-49c5-ad96-d01169bad70b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "881385bf-de76-49c5-ad96-d01169bad70b" (UID: "881385bf-de76-49c5-ad96-d01169bad70b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:57:28.087407 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.087374    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881385bf-de76-49c5-ad96-d01169bad70b-kube-api-access-5m5v7" (OuterVolumeSpecName: "kube-api-access-5m5v7") pod "881385bf-de76-49c5-ad96-d01169bad70b" (UID: "881385bf-de76-49c5-ad96-d01169bad70b"). InnerVolumeSpecName "kube-api-access-5m5v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:57:28.186517 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.186469    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/881385bf-de76-49c5-ad96-d01169bad70b-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:57:28.186517 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.186506    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/881385bf-de76-49c5-ad96-d01169bad70b-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:57:28.186517 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.186518    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m5v7\" (UniqueName: \"kubernetes.io/projected/881385bf-de76-49c5-ad96-d01169bad70b-kube-api-access-5m5v7\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:57:28.186517 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.186527    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/881385bf-de76-49c5-ad96-d01169bad70b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:57:28.509847 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.509793    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9_881385bf-de76-49c5-ad96-d01169bad70b/storage-initializer/1.log"
Jul 27 18:57:28.510382 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.510295    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9_881385bf-de76-49c5-ad96-d01169bad70b/storage-initializer/0.log"
Jul 27 18:57:28.510382 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.510332    2564 generic.go:356] "Generic (PLEG): container finished" podID="881385bf-de76-49c5-ad96-d01169bad70b" containerID="4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f" exitCode=1
Jul 27 18:57:28.510495 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.510405    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"
Jul 27 18:57:28.510495 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.510424    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9" event={"ID":"881385bf-de76-49c5-ad96-d01169bad70b","Type":"ContainerDied","Data":"4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f"}
Jul 27 18:57:28.510605 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.510501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9" event={"ID":"881385bf-de76-49c5-ad96-d01169bad70b","Type":"ContainerDied","Data":"c46afb8d6a9039649a77fdf55ce94bee29962577f24178c539c509e3daff0799"}
Jul 27 18:57:28.510605 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.510540    2564 scope.go:117] "RemoveContainer" containerID="4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f"
Jul 27 18:57:28.521128 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521085    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"]
Jul 27 18:57:28.521464 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521448    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container"
Jul 27 18:57:28.521522 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521466    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container"
Jul 27 18:57:28.521522 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521473    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881385bf-de76-49c5-ad96-d01169bad70b" containerName="storage-initializer"
Jul 27 18:57:28.521522 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521479    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="881385bf-de76-49c5-ad96-d01169bad70b" containerName="storage-initializer"
Jul 27 18:57:28.521522 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521494    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kube-rbac-proxy"
Jul 27 18:57:28.521522 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521500    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kube-rbac-proxy"
Jul 27 18:57:28.521522 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521519    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881385bf-de76-49c5-ad96-d01169bad70b" containerName="storage-initializer"
Jul 27 18:57:28.521522 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521524    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="881385bf-de76-49c5-ad96-d01169bad70b" containerName="storage-initializer"
Jul 27 18:57:28.521860 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521533    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="storage-initializer"
Jul 27 18:57:28.521860 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521539    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="storage-initializer"
Jul 27 18:57:28.521860 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521591    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kube-rbac-proxy"
Jul 27 18:57:28.521860 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521601    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="881385bf-de76-49c5-ad96-d01169bad70b" containerName="storage-initializer"
Jul 27 18:57:28.521860 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521608    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a366b364-f00d-4661-84ac-fc800a48ad58" containerName="kserve-container"
Jul 27 18:57:28.521860 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521614    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="881385bf-de76-49c5-ad96-d01169bad70b" containerName="storage-initializer"
Jul 27 18:57:28.521860 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.521841    2564 scope.go:117] "RemoveContainer" containerID="2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc"
Jul 27 18:57:28.526367 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.526350    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.529234 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.529208    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:57:28.529429 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.529208    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 27 18:57:28.529549 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.529484    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:57:28.529768 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.529737    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:57:28.529905 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.529777    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 27 18:57:28.529905 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.529777    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:57:28.529905 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.529871    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 27 18:57:28.531945 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.531825    2564 scope.go:117] "RemoveContainer" containerID="4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f"
Jul 27 18:57:28.532320 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:57:28.532291    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f\": container with ID starting with 4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f not found: ID does not exist" containerID="4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f"
Jul 27 18:57:28.532435 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.532338    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f"} err="failed to get container status \"4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f\": rpc error: code = NotFound desc = could not find container \"4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f\": container with ID starting with 4232653cda8448b40ec46d3f32a89708cb35cdac75281a7bac61c68cc1a2ec4f not found: ID does not exist"
Jul 27 18:57:28.532435 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.532365    2564 scope.go:117] "RemoveContainer" containerID="2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc"
Jul 27 18:57:28.532813 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:57:28.532787    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc\": container with ID starting with 2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc not found: ID does not exist" containerID="2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc"
Jul 27 18:57:28.532917 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.532825    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc"} err="failed to get container status \"2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc\": rpc error: code = NotFound desc = could not find container \"2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc\": container with ID starting with 2d96e93b890c082062c72033677d737b0ca4baeed38cbcdc3f9f5bf43982c5cc not found: ID does not exist"
Jul 27 18:57:28.535532 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.535511    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"]
Jul 27 18:57:28.555643 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.555605    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"]
Jul 27 18:57:28.559457 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.559419    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-56gl9"]
Jul 27 18:57:28.590664 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.590622    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.590887 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.590683    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.590887 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.590746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrlgj\" (UniqueName: \"kubernetes.io/projected/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-api-access-mrlgj\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.590887 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.590790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a26b2de3-b883-43c4-b3a7-46b880659acf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.590887 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.590839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a26b2de3-b883-43c4-b3a7-46b880659acf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.691351 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.691305    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a26b2de3-b883-43c4-b3a7-46b880659acf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.691351 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.691354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a26b2de3-b883-43c4-b3a7-46b880659acf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.691646 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.691396    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.691646 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.691434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.691646 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.691494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrlgj\" (UniqueName: \"kubernetes.io/projected/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-api-access-mrlgj\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.691830 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.691806    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a26b2de3-b883-43c4-b3a7-46b880659acf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.692138 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.692116    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.692275 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.692252    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.694108 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.694082    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a26b2de3-b883-43c4-b3a7-46b880659acf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.701782 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.701757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrlgj\" (UniqueName: \"kubernetes.io/projected/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-api-access-mrlgj\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.839328 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.839216    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:28.983470 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:28.983427    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"]
Jul 27 18:57:28.987039 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:57:28.987005    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26b2de3_b883_43c4_b3a7_46b880659acf.slice/crio-d07c64de715b79273a6958c6a4a5a7f965765250c03df36cf684c96049b8c073 WatchSource:0}: Error finding container d07c64de715b79273a6958c6a4a5a7f965765250c03df36cf684c96049b8c073: Status 404 returned error can't find the container with id d07c64de715b79273a6958c6a4a5a7f965765250c03df36cf684c96049b8c073
Jul 27 18:57:29.121557 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:29.121474    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="881385bf-de76-49c5-ad96-d01169bad70b" path="/var/lib/kubelet/pods/881385bf-de76-49c5-ad96-d01169bad70b/volumes"
Jul 27 18:57:29.516974 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:29.516933    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" event={"ID":"a26b2de3-b883-43c4-b3a7-46b880659acf","Type":"ContainerStarted","Data":"d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf"}
Jul 27 18:57:29.517525 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:29.517010    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" event={"ID":"a26b2de3-b883-43c4-b3a7-46b880659acf","Type":"ContainerStarted","Data":"d07c64de715b79273a6958c6a4a5a7f965765250c03df36cf684c96049b8c073"}
Jul 27 18:57:30.523751 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:30.523713    2564 generic.go:356] "Generic (PLEG): container finished" podID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerID="d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf" exitCode=0
Jul 27 18:57:30.524232 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:30.523795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" event={"ID":"a26b2de3-b883-43c4-b3a7-46b880659acf","Type":"ContainerDied","Data":"d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf"}
Jul 27 18:57:31.530855 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:31.530818    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" event={"ID":"a26b2de3-b883-43c4-b3a7-46b880659acf","Type":"ContainerStarted","Data":"93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308"}
Jul 27 18:57:31.531307 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:31.530888    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" event={"ID":"a26b2de3-b883-43c4-b3a7-46b880659acf","Type":"ContainerStarted","Data":"4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee"}
Jul 27 18:57:31.531307 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:31.530999    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:31.531307 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:31.531140    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:31.532341 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:31.532314    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:57:31.552208 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:31.552134    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podStartSLOduration=2.496488407 podStartE2EDuration="3.552115962s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.055627555s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:57:31.549855488 +0000 UTC m=+5395.002476635" watchObservedRunningTime="2026-07-27 18:57:31.552115962 +0000 UTC m=+5395.004737108"
Jul 27 18:57:32.535834 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:32.535784    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:57:37.396896 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:37.396772    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:57:37.418287 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:37.405058    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:57:37.418287 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:37.407090    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 18:57:37.418287 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:37.415285    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 18:57:37.540509 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:37.540470    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:57:37.541100 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:37.541064    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:57:47.541965 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:47.541920    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:57:57.541361 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:57:57.541314    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:58:07.541493 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:07.541447    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:58:17.541642 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:17.541587    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:58:27.541646 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:27.541600    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:58:37.541098 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:37.541053    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:58:47.541920 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:47.541869    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 27 18:58:57.542348 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:57.542312    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:58:58.586382 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:58.586289    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"]
Jul 27 18:58:58.586881 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:58.586547    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container" containerID="cri-o://4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee" gracePeriod=30
Jul 27 18:58:58.586881 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:58.586582    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kube-rbac-proxy" containerID="cri-o://93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308" gracePeriod=30
Jul 27 18:58:58.909518 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:58.909424    2564 generic.go:356] "Generic (PLEG): container finished" podID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerID="93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308" exitCode=2
Jul 27 18:58:58.909518 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:58.909500    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" event={"ID":"a26b2de3-b883-43c4-b3a7-46b880659acf","Type":"ContainerDied","Data":"93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308"}
Jul 27 18:58:59.761050 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.761005    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"]
Jul 27 18:58:59.764636 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.764612    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.767702 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.767676    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 27 18:58:59.767825 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.767760    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 27 18:58:59.790934 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.790897    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"]
Jul 27 18:58:59.826796 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.826750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1439a997-0115-4bd7-9218-ffa533a4209b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.827008 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.826850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1439a997-0115-4bd7-9218-ffa533a4209b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.827008 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.826897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1439a997-0115-4bd7-9218-ffa533a4209b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.827008 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.826929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5dwt\" (UniqueName: \"kubernetes.io/projected/1439a997-0115-4bd7-9218-ffa533a4209b-kube-api-access-b5dwt\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.928327 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.928277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1439a997-0115-4bd7-9218-ffa533a4209b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.928327 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.928333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1439a997-0115-4bd7-9218-ffa533a4209b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.928639 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.928451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1439a997-0115-4bd7-9218-ffa533a4209b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.928639 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:58:59.928462    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 27 18:58:59.928639 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.928523    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dwt\" (UniqueName: \"kubernetes.io/projected/1439a997-0115-4bd7-9218-ffa533a4209b-kube-api-access-b5dwt\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.928639 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:58:59.928625    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1439a997-0115-4bd7-9218-ffa533a4209b-proxy-tls podName:1439a997-0115-4bd7-9218-ffa533a4209b nodeName:}" failed. No retries permitted until 2026-07-27 18:59:00.428595547 +0000 UTC m=+5483.881216678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1439a997-0115-4bd7-9218-ffa533a4209b-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m" (UID: "1439a997-0115-4bd7-9218-ffa533a4209b") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 27 18:58:59.928893 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.928869    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1439a997-0115-4bd7-9218-ffa533a4209b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.929259 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.929240    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1439a997-0115-4bd7-9218-ffa533a4209b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:58:59.940454 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:58:59.940420    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dwt\" (UniqueName: \"kubernetes.io/projected/1439a997-0115-4bd7-9218-ffa533a4209b-kube-api-access-b5dwt\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:59:00.433170 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:00.433126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1439a997-0115-4bd7-9218-ffa533a4209b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:59:00.435769 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:00.435742    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1439a997-0115-4bd7-9218-ffa533a4209b-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:59:00.676140 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:00.676083    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:59:00.820556 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:00.820528    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"]
Jul 27 18:59:00.823323 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:59:00.823290    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1439a997_0115_4bd7_9218_ffa533a4209b.slice/crio-77ddc27be12d8c0ed33129589310f9a099223447d08e3fa8643b6057ca39a378 WatchSource:0}: Error finding container 77ddc27be12d8c0ed33129589310f9a099223447d08e3fa8643b6057ca39a378: Status 404 returned error can't find the container with id 77ddc27be12d8c0ed33129589310f9a099223447d08e3fa8643b6057ca39a378
Jul 27 18:59:00.921985 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:00.921946    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m" event={"ID":"1439a997-0115-4bd7-9218-ffa533a4209b","Type":"ContainerStarted","Data":"3ce7321ae75ce5b00491efefd8465cfd655d64ddf82247285133dc9f6bdbad4b"}
Jul 27 18:59:00.922212 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:00.922039    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m" event={"ID":"1439a997-0115-4bd7-9218-ffa533a4209b","Type":"ContainerStarted","Data":"77ddc27be12d8c0ed33129589310f9a099223447d08e3fa8643b6057ca39a378"}
Jul 27 18:59:02.536863 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:02.536811    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Jul 27 18:59:04.344545 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.344516    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:59:04.473084 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.473030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a26b2de3-b883-43c4-b3a7-46b880659acf-kserve-provision-location\") pod \"a26b2de3-b883-43c4-b3a7-46b880659acf\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") "
Jul 27 18:59:04.473408 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.473111    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-cabundle-cert\") pod \"a26b2de3-b883-43c4-b3a7-46b880659acf\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") "
Jul 27 18:59:04.473408 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.473137    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a26b2de3-b883-43c4-b3a7-46b880659acf-proxy-tls\") pod \"a26b2de3-b883-43c4-b3a7-46b880659acf\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") "
Jul 27 18:59:04.473408 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.473157    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-rbac-proxy-sar-config\") pod \"a26b2de3-b883-43c4-b3a7-46b880659acf\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") "
Jul 27 18:59:04.473408 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.473250    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrlgj\" (UniqueName: \"kubernetes.io/projected/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-api-access-mrlgj\") pod \"a26b2de3-b883-43c4-b3a7-46b880659acf\" (UID: \"a26b2de3-b883-43c4-b3a7-46b880659acf\") "
Jul 27 18:59:04.473653 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.473523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26b2de3-b883-43c4-b3a7-46b880659acf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a26b2de3-b883-43c4-b3a7-46b880659acf" (UID: "a26b2de3-b883-43c4-b3a7-46b880659acf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:59:04.473707 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.473648    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a26b2de3-b883-43c4-b3a7-46b880659acf" (UID: "a26b2de3-b883-43c4-b3a7-46b880659acf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:59:04.473707 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.473692    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a26b2de3-b883-43c4-b3a7-46b880659acf" (UID: "a26b2de3-b883-43c4-b3a7-46b880659acf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:59:04.475792 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.475753    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a26b2de3-b883-43c4-b3a7-46b880659acf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a26b2de3-b883-43c4-b3a7-46b880659acf" (UID: "a26b2de3-b883-43c4-b3a7-46b880659acf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:59:04.475934 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.475907    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-api-access-mrlgj" (OuterVolumeSpecName: "kube-api-access-mrlgj") pod "a26b2de3-b883-43c4-b3a7-46b880659acf" (UID: "a26b2de3-b883-43c4-b3a7-46b880659acf"). InnerVolumeSpecName "kube-api-access-mrlgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:59:04.574166 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.574069    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:59:04.574166 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.574100    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrlgj\" (UniqueName: \"kubernetes.io/projected/a26b2de3-b883-43c4-b3a7-46b880659acf-kube-api-access-mrlgj\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:59:04.574166 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.574110    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a26b2de3-b883-43c4-b3a7-46b880659acf-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:59:04.574166 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.574120    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a26b2de3-b883-43c4-b3a7-46b880659acf-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:59:04.574166 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.574130    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a26b2de3-b883-43c4-b3a7-46b880659acf-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:59:04.944534 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.944434    2564 generic.go:356] "Generic (PLEG): container finished" podID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerID="4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee" exitCode=0
Jul 27 18:59:04.944534 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.944498    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"
Jul 27 18:59:04.944534 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.944525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" event={"ID":"a26b2de3-b883-43c4-b3a7-46b880659acf","Type":"ContainerDied","Data":"4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee"}
Jul 27 18:59:04.944834 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.944595    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw" event={"ID":"a26b2de3-b883-43c4-b3a7-46b880659acf","Type":"ContainerDied","Data":"d07c64de715b79273a6958c6a4a5a7f965765250c03df36cf684c96049b8c073"}
Jul 27 18:59:04.944834 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.944632    2564 scope.go:117] "RemoveContainer" containerID="93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308"
Jul 27 18:59:04.946315 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.946296    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_1439a997-0115-4bd7-9218-ffa533a4209b/storage-initializer/0.log"
Jul 27 18:59:04.946453 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.946328    2564 generic.go:356] "Generic (PLEG): container finished" podID="1439a997-0115-4bd7-9218-ffa533a4209b" containerID="3ce7321ae75ce5b00491efefd8465cfd655d64ddf82247285133dc9f6bdbad4b" exitCode=1
Jul 27 18:59:04.946453 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.946405    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m" event={"ID":"1439a997-0115-4bd7-9218-ffa533a4209b","Type":"ContainerDied","Data":"3ce7321ae75ce5b00491efefd8465cfd655d64ddf82247285133dc9f6bdbad4b"}
Jul 27 18:59:04.954115 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.954090    2564 scope.go:117] "RemoveContainer" containerID="4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee"
Jul 27 18:59:04.965337 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.965298    2564 scope.go:117] "RemoveContainer" containerID="d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf"
Jul 27 18:59:04.975512 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.975486    2564 scope.go:117] "RemoveContainer" containerID="93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308"
Jul 27 18:59:04.975848 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:59:04.975824    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308\": container with ID starting with 93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308 not found: ID does not exist" containerID="93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308"
Jul 27 18:59:04.975900 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.975858    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308"} err="failed to get container status \"93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308\": rpc error: code = NotFound desc = could not find container \"93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308\": container with ID starting with 93206850105d3a819ed574d447684847f04a58f04521e366b4acb8a32b917308 not found: ID does not exist"
Jul 27 18:59:04.975900 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.975881    2564 scope.go:117] "RemoveContainer" containerID="4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee"
Jul 27 18:59:04.976154 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:59:04.976137    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee\": container with ID starting with 4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee not found: ID does not exist" containerID="4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee"
Jul 27 18:59:04.976227 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.976156    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee"} err="failed to get container status \"4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee\": rpc error: code = NotFound desc = could not find container \"4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee\": container with ID starting with 4fd8f431d4a342979ac85667206555671d3c0df53b485455cfd91ea6b8ea3fee not found: ID does not exist"
Jul 27 18:59:04.976227 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.976174    2564 scope.go:117] "RemoveContainer" containerID="d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf"
Jul 27 18:59:04.976428 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:59:04.976410    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf\": container with ID starting with d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf not found: ID does not exist" containerID="d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf"
Jul 27 18:59:04.976467 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.976440    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf"} err="failed to get container status \"d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf\": rpc error: code = NotFound desc = could not find container \"d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf\": container with ID starting with d476d7f5003b2a043b8712a10923fc810ace0de2d24e804e003a104921ece0bf not found: ID does not exist"
Jul 27 18:59:04.990577 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.990546    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"]
Jul 27 18:59:04.992781 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:04.992755    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-2s8sw"]
Jul 27 18:59:05.120711 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:05.120675    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" path="/var/lib/kubelet/pods/a26b2de3-b883-43c4-b3a7-46b880659acf/volumes"
Jul 27 18:59:05.954085 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:05.954052    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_1439a997-0115-4bd7-9218-ffa533a4209b/storage-initializer/0.log"
Jul 27 18:59:05.954524 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:05.954133    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m" event={"ID":"1439a997-0115-4bd7-9218-ffa533a4209b","Type":"ContainerStarted","Data":"7a664c84c5312b8f0223c642878c2d58fefb9823737de49e557311f7e9bd2330"}
Jul 27 18:59:08.972465 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:08.972433    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_1439a997-0115-4bd7-9218-ffa533a4209b/storage-initializer/1.log"
Jul 27 18:59:08.973134 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:08.972875    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_1439a997-0115-4bd7-9218-ffa533a4209b/storage-initializer/0.log"
Jul 27 18:59:08.973134 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:08.972920    2564 generic.go:356] "Generic (PLEG): container finished" podID="1439a997-0115-4bd7-9218-ffa533a4209b" containerID="7a664c84c5312b8f0223c642878c2d58fefb9823737de49e557311f7e9bd2330" exitCode=1
Jul 27 18:59:08.973134 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:08.972992    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m" event={"ID":"1439a997-0115-4bd7-9218-ffa533a4209b","Type":"ContainerDied","Data":"7a664c84c5312b8f0223c642878c2d58fefb9823737de49e557311f7e9bd2330"}
Jul 27 18:59:08.973134 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:08.973093    2564 scope.go:117] "RemoveContainer" containerID="3ce7321ae75ce5b00491efefd8465cfd655d64ddf82247285133dc9f6bdbad4b"
Jul 27 18:59:08.973537 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:08.973520    2564 scope.go:117] "RemoveContainer" containerID="3ce7321ae75ce5b00491efefd8465cfd655d64ddf82247285133dc9f6bdbad4b"
Jul 27 18:59:08.985260 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:59:08.985218    2564 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_kserve-ci-e2e-test_1439a997-0115-4bd7-9218-ffa533a4209b_0 in pod sandbox 77ddc27be12d8c0ed33129589310f9a099223447d08e3fa8643b6057ca39a378 from index: no such id: '3ce7321ae75ce5b00491efefd8465cfd655d64ddf82247285133dc9f6bdbad4b'" containerID="3ce7321ae75ce5b00491efefd8465cfd655d64ddf82247285133dc9f6bdbad4b"
Jul 27 18:59:08.985434 ip-10-0-129-58 kubenswrapper[2564]: E0727 18:59:08.985257    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_kserve-ci-e2e-test(1439a997-0115-4bd7-9218-ffa533a4209b)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m" podUID="1439a997-0115-4bd7-9218-ffa533a4209b"
Jul 27 18:59:08.985434 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:08.985268    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce7321ae75ce5b00491efefd8465cfd655d64ddf82247285133dc9f6bdbad4b"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_kserve-ci-e2e-test_1439a997-0115-4bd7-9218-ffa533a4209b_0 in pod sandbox 77ddc27be12d8c0ed33129589310f9a099223447d08e3fa8643b6057ca39a378 from index: no such id: '3ce7321ae75ce5b00491efefd8465cfd655d64ddf82247285133dc9f6bdbad4b'"
Jul 27 18:59:09.662463 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:09.662425    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"]
Jul 27 18:59:09.979749 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:09.979719    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_1439a997-0115-4bd7-9218-ffa533a4209b/storage-initializer/1.log"
Jul 27 18:59:10.126070 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.126044    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_1439a997-0115-4bd7-9218-ffa533a4209b/storage-initializer/1.log"
Jul 27 18:59:10.129827 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.129802    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:59:10.228565 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.228499    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1439a997-0115-4bd7-9218-ffa533a4209b-proxy-tls\") pod \"1439a997-0115-4bd7-9218-ffa533a4209b\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") "
Jul 27 18:59:10.228765 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.228585    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5dwt\" (UniqueName: \"kubernetes.io/projected/1439a997-0115-4bd7-9218-ffa533a4209b-kube-api-access-b5dwt\") pod \"1439a997-0115-4bd7-9218-ffa533a4209b\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") "
Jul 27 18:59:10.228765 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.228688    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1439a997-0115-4bd7-9218-ffa533a4209b-kube-rbac-proxy-sar-config\") pod \"1439a997-0115-4bd7-9218-ffa533a4209b\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") "
Jul 27 18:59:10.228846 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.228742    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1439a997-0115-4bd7-9218-ffa533a4209b-kserve-provision-location\") pod \"1439a997-0115-4bd7-9218-ffa533a4209b\" (UID: \"1439a997-0115-4bd7-9218-ffa533a4209b\") "
Jul 27 18:59:10.229070 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.229042    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1439a997-0115-4bd7-9218-ffa533a4209b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1439a997-0115-4bd7-9218-ffa533a4209b" (UID: "1439a997-0115-4bd7-9218-ffa533a4209b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:59:10.229070 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.229068    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1439a997-0115-4bd7-9218-ffa533a4209b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1439a997-0115-4bd7-9218-ffa533a4209b" (UID: "1439a997-0115-4bd7-9218-ffa533a4209b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:59:10.230808 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.230748    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1439a997-0115-4bd7-9218-ffa533a4209b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1439a997-0115-4bd7-9218-ffa533a4209b" (UID: "1439a997-0115-4bd7-9218-ffa533a4209b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:59:10.230910 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.230838    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1439a997-0115-4bd7-9218-ffa533a4209b-kube-api-access-b5dwt" (OuterVolumeSpecName: "kube-api-access-b5dwt") pod "1439a997-0115-4bd7-9218-ffa533a4209b" (UID: "1439a997-0115-4bd7-9218-ffa533a4209b"). InnerVolumeSpecName "kube-api-access-b5dwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:59:10.330029 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.329987    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5dwt\" (UniqueName: \"kubernetes.io/projected/1439a997-0115-4bd7-9218-ffa533a4209b-kube-api-access-b5dwt\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:59:10.330029 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.330022    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1439a997-0115-4bd7-9218-ffa533a4209b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:59:10.330029 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.330030    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1439a997-0115-4bd7-9218-ffa533a4209b-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:59:10.330029 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.330039    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1439a997-0115-4bd7-9218-ffa533a4209b-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 18:59:10.764702 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.764663    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"]
Jul 27 18:59:10.765138 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765111    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kube-rbac-proxy"
Jul 27 18:59:10.765138 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765140    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kube-rbac-proxy"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765156    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1439a997-0115-4bd7-9218-ffa533a4209b" containerName="storage-initializer"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765166    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1439a997-0115-4bd7-9218-ffa533a4209b" containerName="storage-initializer"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765197    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="storage-initializer"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765208    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="storage-initializer"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765220    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765228    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765249    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1439a997-0115-4bd7-9218-ffa533a4209b" containerName="storage-initializer"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765257    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1439a997-0115-4bd7-9218-ffa533a4209b" containerName="storage-initializer"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765347    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1439a997-0115-4bd7-9218-ffa533a4209b" containerName="storage-initializer"
Jul 27 18:59:10.765368 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765359    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kube-rbac-proxy"
Jul 27 18:59:10.765852 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765374    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1439a997-0115-4bd7-9218-ffa533a4209b" containerName="storage-initializer"
Jul 27 18:59:10.765852 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.765385    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a26b2de3-b883-43c4-b3a7-46b880659acf" containerName="kserve-container"
Jul 27 18:59:10.768938 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.768911    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:10.771964 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.771929    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 27 18:59:10.772110 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.771940    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:59:10.772110 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.771942    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 27 18:59:10.779213 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.779164    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"]
Jul 27 18:59:10.935345 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.935300    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc547a7e-1206-413d-aa30-c29905eda56b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:10.935547 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.935379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:10.935547 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.935422    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:10.935547 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.935455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvsc\" (UniqueName: \"kubernetes.io/projected/cc547a7e-1206-413d-aa30-c29905eda56b-kube-api-access-fpvsc\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:10.935547 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.935526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc547a7e-1206-413d-aa30-c29905eda56b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:10.985725 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.985694    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m_1439a997-0115-4bd7-9218-ffa533a4209b/storage-initializer/1.log"
Jul 27 18:59:10.986153 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.985804    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"
Jul 27 18:59:10.986153 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.985795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m" event={"ID":"1439a997-0115-4bd7-9218-ffa533a4209b","Type":"ContainerDied","Data":"77ddc27be12d8c0ed33129589310f9a099223447d08e3fa8643b6057ca39a378"}
Jul 27 18:59:10.986153 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:10.985952    2564 scope.go:117] "RemoveContainer" containerID="7a664c84c5312b8f0223c642878c2d58fefb9823737de49e557311f7e9bd2330"
Jul 27 18:59:11.022710 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.022621    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"]
Jul 27 18:59:11.027498 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.027464    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-snq9m"]
Jul 27 18:59:11.036091 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.036059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc547a7e-1206-413d-aa30-c29905eda56b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.036277 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.036123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc547a7e-1206-413d-aa30-c29905eda56b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.036277 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.036154    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.036277 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.036208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.036277 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.036257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvsc\" (UniqueName: \"kubernetes.io/projected/cc547a7e-1206-413d-aa30-c29905eda56b-kube-api-access-fpvsc\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.036611 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.036589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc547a7e-1206-413d-aa30-c29905eda56b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.036967 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.036948    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.037031 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.036968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.038885 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.038854    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc547a7e-1206-413d-aa30-c29905eda56b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.045113 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.045086    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvsc\" (UniqueName: \"kubernetes.io/projected/cc547a7e-1206-413d-aa30-c29905eda56b-kube-api-access-fpvsc\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.081964 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.081926    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:11.120860 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.120824    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1439a997-0115-4bd7-9218-ffa533a4209b" path="/var/lib/kubelet/pods/1439a997-0115-4bd7-9218-ffa533a4209b/volumes"
Jul 27 18:59:11.224673 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.224636    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"]
Jul 27 18:59:11.230007 ip-10-0-129-58 kubenswrapper[2564]: W0727 18:59:11.229962    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc547a7e_1206_413d_aa30_c29905eda56b.slice/crio-7cc7ffdb1e611619a456382a1324ee34e8dfc562d07e90b559e2822a62d93745 WatchSource:0}: Error finding container 7cc7ffdb1e611619a456382a1324ee34e8dfc562d07e90b559e2822a62d93745: Status 404 returned error can't find the container with id 7cc7ffdb1e611619a456382a1324ee34e8dfc562d07e90b559e2822a62d93745
Jul 27 18:59:11.994667 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.994627    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" event={"ID":"cc547a7e-1206-413d-aa30-c29905eda56b","Type":"ContainerStarted","Data":"176dca49b1a8b7c666aa91f516fa4be0a00cc27e183ea95eeb547470b3a807a6"}
Jul 27 18:59:11.995212 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:11.994699    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" event={"ID":"cc547a7e-1206-413d-aa30-c29905eda56b","Type":"ContainerStarted","Data":"7cc7ffdb1e611619a456382a1324ee34e8dfc562d07e90b559e2822a62d93745"}
Jul 27 18:59:13.000831 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:13.000793    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc547a7e-1206-413d-aa30-c29905eda56b" containerID="176dca49b1a8b7c666aa91f516fa4be0a00cc27e183ea95eeb547470b3a807a6" exitCode=0
Jul 27 18:59:13.001347 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:13.000843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" event={"ID":"cc547a7e-1206-413d-aa30-c29905eda56b","Type":"ContainerDied","Data":"176dca49b1a8b7c666aa91f516fa4be0a00cc27e183ea95eeb547470b3a807a6"}
Jul 27 18:59:14.008479 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:14.008444    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" event={"ID":"cc547a7e-1206-413d-aa30-c29905eda56b","Type":"ContainerStarted","Data":"0a9d8d9e1fadc35fd04325d6f39ce2533616ba6d30e04362e85d7198967b87fe"}
Jul 27 18:59:14.008949 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:14.008516    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" event={"ID":"cc547a7e-1206-413d-aa30-c29905eda56b","Type":"ContainerStarted","Data":"c32c9cfd2d8c7ba8032c863bc0f8d6108e27b291c724017be71687f2fffdbd64"}
Jul 27 18:59:14.008949 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:14.008695    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:14.008949 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:14.008842    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:14.010080 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:14.010048    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 18:59:14.029200 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:14.029129    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podStartSLOduration=2.9821080970000002 podStartE2EDuration="4.029109826s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.047001729s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:59:14.027925066 +0000 UTC m=+5497.480546212" watchObservedRunningTime="2026-07-27 18:59:14.029109826 +0000 UTC m=+5497.481730972"
Jul 27 18:59:15.013776 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:15.013728    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 18:59:20.019112 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:20.019078    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 18:59:20.019681 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:20.019646    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 18:59:30.020469 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:30.020424    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 18:59:40.020383 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:40.020327    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 18:59:50.020240 ip-10-0-129-58 kubenswrapper[2564]: I0727 18:59:50.020170    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 19:00:00.019977 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:00.019925    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 19:00:10.019971 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:10.019921    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 19:00:20.020658 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:20.020606    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 19:00:30.019655 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:30.019553    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 19:00:35.119915 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:35.119882    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 19:00:40.859222 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:40.859162    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"]
Jul 27 19:00:40.859818 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:40.859481    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" containerID="cri-o://c32c9cfd2d8c7ba8032c863bc0f8d6108e27b291c724017be71687f2fffdbd64" gracePeriod=30
Jul 27 19:00:40.859818 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:40.859531    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kube-rbac-proxy" containerID="cri-o://0a9d8d9e1fadc35fd04325d6f39ce2533616ba6d30e04362e85d7198967b87fe" gracePeriod=30
Jul 27 19:00:41.399556 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.399517    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc547a7e-1206-413d-aa30-c29905eda56b" containerID="0a9d8d9e1fadc35fd04325d6f39ce2533616ba6d30e04362e85d7198967b87fe" exitCode=2
Jul 27 19:00:41.399753 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.399593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" event={"ID":"cc547a7e-1206-413d-aa30-c29905eda56b","Type":"ContainerDied","Data":"0a9d8d9e1fadc35fd04325d6f39ce2533616ba6d30e04362e85d7198967b87fe"}
Jul 27 19:00:41.956371 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.956329    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"]
Jul 27 19:00:41.959951 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.959930    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:41.961439 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.961404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:41.961602 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.961474    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhqc\" (UniqueName: \"kubernetes.io/projected/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-api-access-7fhqc\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:41.961602 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.961528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:41.961686 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.961634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:41.964404 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.964375    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 27 19:00:41.964576 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.964428    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 27 19:00:41.976329 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:41.976294    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"]
Jul 27 19:00:42.062569 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.062520    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:42.062569 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.062574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:42.062859 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.062622    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhqc\" (UniqueName: \"kubernetes.io/projected/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-api-access-7fhqc\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:42.062859 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.062748    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:42.062967 ip-10-0-129-58 kubenswrapper[2564]: E0727 19:00:42.062927    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 27 19:00:42.063024 ip-10-0-129-58 kubenswrapper[2564]: E0727 19:00:42.063018    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-proxy-tls podName:cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4 nodeName:}" failed. No retries permitted until 2026-07-27 19:00:42.562995096 +0000 UTC m=+5586.015616228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk" (UID: "cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 27 19:00:42.063081 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.063060    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:42.063400 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.063381    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:42.073592 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.073552    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhqc\" (UniqueName: \"kubernetes.io/projected/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-api-access-7fhqc\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:42.568610 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.568549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:42.571159 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.571130    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:42.871075 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:42.870969    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:43.015681 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:43.015657    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"]
Jul 27 19:00:43.018280 ip-10-0-129-58 kubenswrapper[2564]: W0727 19:00:43.018250    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf46cb4f_3125_4fb6_bf80_8889b0f7e2c4.slice/crio-215cf062ec3c9ef25e317a7ddc3c1cbaa37922f5ace91fd2092081d9c91fde41 WatchSource:0}: Error finding container 215cf062ec3c9ef25e317a7ddc3c1cbaa37922f5ace91fd2092081d9c91fde41: Status 404 returned error can't find the container with id 215cf062ec3c9ef25e317a7ddc3c1cbaa37922f5ace91fd2092081d9c91fde41
Jul 27 19:00:43.412071 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:43.412033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk" event={"ID":"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4","Type":"ContainerStarted","Data":"c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770"}
Jul 27 19:00:43.412287 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:43.412095    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk" event={"ID":"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4","Type":"ContainerStarted","Data":"215cf062ec3c9ef25e317a7ddc3c1cbaa37922f5ace91fd2092081d9c91fde41"}
Jul 27 19:00:45.014472 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:45.014429    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Jul 27 19:00:45.116443 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:45.116366    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 27 19:00:46.431832 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.431797    2564 generic.go:356] "Generic (PLEG): container finished" podID="cc547a7e-1206-413d-aa30-c29905eda56b" containerID="c32c9cfd2d8c7ba8032c863bc0f8d6108e27b291c724017be71687f2fffdbd64" exitCode=0
Jul 27 19:00:46.432295 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.431833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" event={"ID":"cc547a7e-1206-413d-aa30-c29905eda56b","Type":"ContainerDied","Data":"c32c9cfd2d8c7ba8032c863bc0f8d6108e27b291c724017be71687f2fffdbd64"}
Jul 27 19:00:46.516506 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.516476    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 19:00:46.606628 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.606514    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpvsc\" (UniqueName: \"kubernetes.io/projected/cc547a7e-1206-413d-aa30-c29905eda56b-kube-api-access-fpvsc\") pod \"cc547a7e-1206-413d-aa30-c29905eda56b\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") "
Jul 27 19:00:46.606628 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.606566    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-kube-rbac-proxy-sar-config\") pod \"cc547a7e-1206-413d-aa30-c29905eda56b\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") "
Jul 27 19:00:46.606628 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.606598    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-cabundle-cert\") pod \"cc547a7e-1206-413d-aa30-c29905eda56b\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") "
Jul 27 19:00:46.606628 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.606631    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc547a7e-1206-413d-aa30-c29905eda56b-proxy-tls\") pod \"cc547a7e-1206-413d-aa30-c29905eda56b\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") "
Jul 27 19:00:46.606998 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.606665    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc547a7e-1206-413d-aa30-c29905eda56b-kserve-provision-location\") pod \"cc547a7e-1206-413d-aa30-c29905eda56b\" (UID: \"cc547a7e-1206-413d-aa30-c29905eda56b\") "
Jul 27 19:00:46.607052 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.607019    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc547a7e-1206-413d-aa30-c29905eda56b" (UID: "cc547a7e-1206-413d-aa30-c29905eda56b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:00:46.607052 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.607030    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc547a7e-1206-413d-aa30-c29905eda56b" (UID: "cc547a7e-1206-413d-aa30-c29905eda56b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:00:46.607052 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.607043    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc547a7e-1206-413d-aa30-c29905eda56b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc547a7e-1206-413d-aa30-c29905eda56b" (UID: "cc547a7e-1206-413d-aa30-c29905eda56b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:00:46.608847 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.608812    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc547a7e-1206-413d-aa30-c29905eda56b-kube-api-access-fpvsc" (OuterVolumeSpecName: "kube-api-access-fpvsc") pod "cc547a7e-1206-413d-aa30-c29905eda56b" (UID: "cc547a7e-1206-413d-aa30-c29905eda56b"). InnerVolumeSpecName "kube-api-access-fpvsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:00:46.608968 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.608890    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc547a7e-1206-413d-aa30-c29905eda56b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc547a7e-1206-413d-aa30-c29905eda56b" (UID: "cc547a7e-1206-413d-aa30-c29905eda56b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:00:46.708248 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.708202    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-cabundle-cert\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:00:46.708248 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.708239    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc547a7e-1206-413d-aa30-c29905eda56b-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:00:46.708248 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.708250    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc547a7e-1206-413d-aa30-c29905eda56b-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:00:46.708248 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.708259    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvsc\" (UniqueName: \"kubernetes.io/projected/cc547a7e-1206-413d-aa30-c29905eda56b-kube-api-access-fpvsc\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:00:46.708541 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:46.708270    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc547a7e-1206-413d-aa30-c29905eda56b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:00:47.438886 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.438787    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"
Jul 27 19:00:47.438886 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.438801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7" event={"ID":"cc547a7e-1206-413d-aa30-c29905eda56b","Type":"ContainerDied","Data":"7cc7ffdb1e611619a456382a1324ee34e8dfc562d07e90b559e2822a62d93745"}
Jul 27 19:00:47.439418 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.438888    2564 scope.go:117] "RemoveContainer" containerID="0a9d8d9e1fadc35fd04325d6f39ce2533616ba6d30e04362e85d7198967b87fe"
Jul 27 19:00:47.440540 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.440522    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk_cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4/storage-initializer/0.log"
Jul 27 19:00:47.440680 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.440557    2564 generic.go:356] "Generic (PLEG): container finished" podID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" containerID="c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770" exitCode=1
Jul 27 19:00:47.440680 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.440644    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk" event={"ID":"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4","Type":"ContainerDied","Data":"c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770"}
Jul 27 19:00:47.451130 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.451104    2564 scope.go:117] "RemoveContainer" containerID="c32c9cfd2d8c7ba8032c863bc0f8d6108e27b291c724017be71687f2fffdbd64"
Jul 27 19:00:47.460289 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.460242    2564 scope.go:117] "RemoveContainer" containerID="176dca49b1a8b7c666aa91f516fa4be0a00cc27e183ea95eeb547470b3a807a6"
Jul 27 19:00:47.461539 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.461485    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"]
Jul 27 19:00:47.467582 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:47.467555    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-6dcf7"]
Jul 27 19:00:48.447221 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:48.447173    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk_cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4/storage-initializer/0.log"
Jul 27 19:00:48.447684 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:48.447286    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk" event={"ID":"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4","Type":"ContainerStarted","Data":"fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb"}
Jul 27 19:00:49.120440 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:49.120400    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" path="/var/lib/kubelet/pods/cc547a7e-1206-413d-aa30-c29905eda56b/volumes"
Jul 27 19:00:51.884109 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:51.884062    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"]
Jul 27 19:00:51.884584 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:51.884320    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk" podUID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" containerName="storage-initializer" containerID="cri-o://fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb" gracePeriod=30
Jul 27 19:00:53.439334 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.439304    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk_cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4/storage-initializer/1.log"
Jul 27 19:00:53.439797 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.439776    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk_cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4/storage-initializer/0.log"
Jul 27 19:00:53.442130 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.442105    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:53.464247 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.464213    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fhqc\" (UniqueName: \"kubernetes.io/projected/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-api-access-7fhqc\") pod \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") "
Jul 27 19:00:53.464247 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.464252    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-rbac-proxy-sar-config\") pod \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") "
Jul 27 19:00:53.464557 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.464320    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-proxy-tls\") pod \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") "
Jul 27 19:00:53.464557 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.464388    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kserve-provision-location\") pod \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\" (UID: \"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4\") "
Jul 27 19:00:53.464729 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.464700    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" (UID: "cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:00:53.464806 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.464726    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" (UID: "cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:00:53.466813 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.466780    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" (UID: "cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:00:53.466813 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.466801    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-api-access-7fhqc" (OuterVolumeSpecName: "kube-api-access-7fhqc") pod "cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" (UID: "cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4"). InnerVolumeSpecName "kube-api-access-7fhqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:00:53.472909 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.472883    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk_cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4/storage-initializer/1.log"
Jul 27 19:00:53.473401 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.473383    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk_cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4/storage-initializer/0.log"
Jul 27 19:00:53.473502 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.473428    2564 generic.go:356] "Generic (PLEG): container finished" podID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" containerID="fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb" exitCode=1
Jul 27 19:00:53.473553 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.473504    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"
Jul 27 19:00:53.473553 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.473506    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk" event={"ID":"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4","Type":"ContainerDied","Data":"fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb"}
Jul 27 19:00:53.473623 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.473570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk" event={"ID":"cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4","Type":"ContainerDied","Data":"215cf062ec3c9ef25e317a7ddc3c1cbaa37922f5ace91fd2092081d9c91fde41"}
Jul 27 19:00:53.473623 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.473601    2564 scope.go:117] "RemoveContainer" containerID="fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb"
Jul 27 19:00:53.487755 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.487731    2564 scope.go:117] "RemoveContainer" containerID="c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770"
Jul 27 19:00:53.496286 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.496264    2564 scope.go:117] "RemoveContainer" containerID="fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb"
Jul 27 19:00:53.496594 ip-10-0-129-58 kubenswrapper[2564]: E0727 19:00:53.496570    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb\": container with ID starting with fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb not found: ID does not exist" containerID="fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb"
Jul 27 19:00:53.496641 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.496605    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb"} err="failed to get container status \"fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb\": rpc error: code = NotFound desc = could not find container \"fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb\": container with ID starting with fc8fdab8aa15ab3b08017bb7ef4b3d1da53251d584f15e347cf64fa5efbc13fb not found: ID does not exist"
Jul 27 19:00:53.496641 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.496624    2564 scope.go:117] "RemoveContainer" containerID="c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770"
Jul 27 19:00:53.496888 ip-10-0-129-58 kubenswrapper[2564]: E0727 19:00:53.496872    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770\": container with ID starting with c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770 not found: ID does not exist" containerID="c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770"
Jul 27 19:00:53.496931 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.496892    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770"} err="failed to get container status \"c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770\": rpc error: code = NotFound desc = could not find container \"c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770\": container with ID starting with c128e0de03d9f93abb291e2a05d002d894911ecc9cec80f71fd0b3a0948dc770 not found: ID does not exist"
Jul 27 19:00:53.515014 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.514977    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"]
Jul 27 19:00:53.519679 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.519645    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-86lvk"]
Jul 27 19:00:53.565615 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.565515    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-proxy-tls\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:00:53.565615 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.565549    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kserve-provision-location\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:00:53.565615 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.565582    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fhqc\" (UniqueName: \"kubernetes.io/projected/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-api-access-7fhqc\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:00:53.565615 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.565594    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:00:53.602808 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.602759    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jcrqv/must-gather-n9bfk"]
Jul 27 19:00:53.603162 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603145    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" containerName="storage-initializer"
Jul 27 19:00:53.603282 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603166    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" containerName="storage-initializer"
Jul 27 19:00:53.603282 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603201    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kube-rbac-proxy"
Jul 27 19:00:53.603282 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603209    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kube-rbac-proxy"
Jul 27 19:00:53.603282 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603232    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" containerName="storage-initializer"
Jul 27 19:00:53.603282 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603241    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" containerName="storage-initializer"
Jul 27 19:00:53.603282 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603257    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="storage-initializer"
Jul 27 19:00:53.603282 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603265    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="storage-initializer"
Jul 27 19:00:53.603622 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603285    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container"
Jul 27 19:00:53.603622 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603295    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container"
Jul 27 19:00:53.603622 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603382    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" containerName="storage-initializer"
Jul 27 19:00:53.603622 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603393    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kserve-container"
Jul 27 19:00:53.603622 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603404    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc547a7e-1206-413d-aa30-c29905eda56b" containerName="kube-rbac-proxy"
Jul 27 19:00:53.603622 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.603414    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" containerName="storage-initializer"
Jul 27 19:00:53.608080 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.608053    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:00:53.610915 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.610882    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jcrqv\"/\"kube-root-ca.crt\""
Jul 27 19:00:53.610915 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.610899    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jcrqv\"/\"openshift-service-ca.crt\""
Jul 27 19:00:53.614451 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.614424    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jcrqv/must-gather-n9bfk"]
Jul 27 19:00:53.666958 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.666919    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-must-gather-output\") pod \"must-gather-n9bfk\" (UID: \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\") " pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:00:53.666958 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.666969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4x8z\" (UniqueName: \"kubernetes.io/projected/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-kube-api-access-x4x8z\") pod \"must-gather-n9bfk\" (UID: \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\") " pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:00:53.768167 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.768122    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-must-gather-output\") pod \"must-gather-n9bfk\" (UID: \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\") " pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:00:53.768420 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.768206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4x8z\" (UniqueName: \"kubernetes.io/projected/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-kube-api-access-x4x8z\") pod \"must-gather-n9bfk\" (UID: \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\") " pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:00:53.768525 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.768501    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-must-gather-output\") pod \"must-gather-n9bfk\" (UID: \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\") " pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:00:53.776164 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.776138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4x8z\" (UniqueName: \"kubernetes.io/projected/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-kube-api-access-x4x8z\") pod \"must-gather-n9bfk\" (UID: \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\") " pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:00:53.919543 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:53.919436    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:00:54.058595 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:54.058563    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jcrqv/must-gather-n9bfk"]
Jul 27 19:00:54.061090 ip-10-0-129-58 kubenswrapper[2564]: W0727 19:00:54.061059    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ec1e41c_5ee8_4b4c_bbd7_4a3cac3cbfaf.slice/crio-fbc50d065f336fcee0432467d401f55909d89ce93691d2e4d914f65930811c1b WatchSource:0}: Error finding container fbc50d065f336fcee0432467d401f55909d89ce93691d2e4d914f65930811c1b: Status 404 returned error can't find the container with id fbc50d065f336fcee0432467d401f55909d89ce93691d2e4d914f65930811c1b
Jul 27 19:00:54.062803 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:54.062784    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:00:54.478946 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:54.478908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jcrqv/must-gather-n9bfk" event={"ID":"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf","Type":"ContainerStarted","Data":"fbc50d065f336fcee0432467d401f55909d89ce93691d2e4d914f65930811c1b"}
Jul 27 19:00:55.119653 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:00:55.119618    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4" path="/var/lib/kubelet/pods/cf46cb4f-3125-4fb6-bf80-8889b0f7e2c4/volumes"
Jul 27 19:01:01.525408 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:01.525359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jcrqv/must-gather-n9bfk" event={"ID":"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf","Type":"ContainerStarted","Data":"c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6"}
Jul 27 19:01:01.525879 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:01.525416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jcrqv/must-gather-n9bfk" event={"ID":"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf","Type":"ContainerStarted","Data":"dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598"}
Jul 27 19:01:01.544429 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:01.544358    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jcrqv/must-gather-n9bfk" podStartSLOduration=1.515150679 podStartE2EDuration="8.544335358s" totalImagesPullingTime="7.029184679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:01:01.542260285 +0000 UTC m=+5604.994881435" watchObservedRunningTime="2026-07-27 19:01:01.544335358 +0000 UTC m=+5604.996956611"
Jul 27 19:01:28.665381 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:28.665344    2564 generic.go:356] "Generic (PLEG): container finished" podID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" containerID="dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598" exitCode=0
Jul 27 19:01:28.665937 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:28.665400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jcrqv/must-gather-n9bfk" event={"ID":"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf","Type":"ContainerDied","Data":"dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598"}
Jul 27 19:01:28.665937 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:28.665715    2564 scope.go:117] "RemoveContainer" containerID="dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598"
Jul 27 19:01:29.227544 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:29.227508    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jcrqv_must-gather-n9bfk_6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf/gather/0.log"
Jul 27 19:01:32.772216 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:32.772131    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ggttg_3b376d75-6fb9-48ea-8621-e43b18a91e98/global-pull-secret-syncer/0.log"
Jul 27 19:01:32.922174 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:32.922132    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fl5bn_0b95fb06-43f5-441b-b46c-60a5c2cc0d9e/konnectivity-agent/0.log"
Jul 27 19:01:33.009136 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:33.009096    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-58.ec2.internal_ebc1b9d3588ffefd5cf60f36ae3035ca/haproxy/0.log"
Jul 27 19:01:34.810494 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:34.810454    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jcrqv/must-gather-n9bfk"]
Jul 27 19:01:34.811074 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:34.810662    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-jcrqv/must-gather-n9bfk" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" containerName="copy" containerID="cri-o://c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6" gracePeriod=2
Jul 27 19:01:34.813285 ip-10-0-129-58 kubenswrapper[2564]: E0727 19:01:34.813243    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-n9bfk\" is forbidden: User \"system:node:ip-10-0-129-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jcrqv\": no relationship found between node 'ip-10-0-129-58.ec2.internal' and this object" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:01:34.814626 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:34.814600    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jcrqv/must-gather-n9bfk"]
Jul 27 19:01:35.048437 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.048410    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jcrqv_must-gather-n9bfk_6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf/copy/0.log"
Jul 27 19:01:35.049999 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.049976    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:01:35.052416 ip-10-0-129-58 kubenswrapper[2564]: E0727 19:01:35.052387    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-n9bfk\" is forbidden: User \"system:node:ip-10-0-129-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jcrqv\": no relationship found between node 'ip-10-0-129-58.ec2.internal' and this object" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:01:35.158333 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.158233    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-must-gather-output\") pod \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\" (UID: \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\") "
Jul 27 19:01:35.158333 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.158312    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4x8z\" (UniqueName: \"kubernetes.io/projected/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-kube-api-access-x4x8z\") pod \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\" (UID: \"6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf\") "
Jul 27 19:01:35.159541 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.159516    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" (UID: "6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:01:35.160676 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.160648    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-kube-api-access-x4x8z" (OuterVolumeSpecName: "kube-api-access-x4x8z") pod "6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" (UID: "6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf"). InnerVolumeSpecName "kube-api-access-x4x8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:01:35.259126 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.259076    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4x8z\" (UniqueName: \"kubernetes.io/projected/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-kube-api-access-x4x8z\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:01:35.259126 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.259116    2564 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf-must-gather-output\") on node \"ip-10-0-129-58.ec2.internal\" DevicePath \"\""
Jul 27 19:01:35.698493 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.698463    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jcrqv_must-gather-n9bfk_6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf/copy/0.log"
Jul 27 19:01:35.698921 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.698894    2564 generic.go:356] "Generic (PLEG): container finished" podID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" containerID="c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6" exitCode=143
Jul 27 19:01:35.698987 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.698956    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jcrqv/must-gather-n9bfk"
Jul 27 19:01:35.698987 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.698967    2564 scope.go:117] "RemoveContainer" containerID="c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6"
Jul 27 19:01:35.707795 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.707773    2564 scope.go:117] "RemoveContainer" containerID="dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598"
Jul 27 19:01:35.720588 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.720566    2564 scope.go:117] "RemoveContainer" containerID="c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6"
Jul 27 19:01:35.720954 ip-10-0-129-58 kubenswrapper[2564]: E0727 19:01:35.720930    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6\": container with ID starting with c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6 not found: ID does not exist" containerID="c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6"
Jul 27 19:01:35.721044 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.720963    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6"} err="failed to get container status \"c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6\": rpc error: code = NotFound desc = could not find container \"c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6\": container with ID starting with c7ac8b60153c030bee4cbd2db3508c9ea8377eea8f698d1caee98f46d26763d6 not found: ID does not exist"
Jul 27 19:01:35.721044 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.720985    2564 scope.go:117] "RemoveContainer" containerID="dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598"
Jul 27 19:01:35.721282 ip-10-0-129-58 kubenswrapper[2564]: E0727 19:01:35.721265    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598\": container with ID starting with dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598 not found: ID does not exist" containerID="dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598"
Jul 27 19:01:35.721348 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:35.721287    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598"} err="failed to get container status \"dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598\": rpc error: code = NotFound desc = could not find container \"dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598\": container with ID starting with dcb98a0e43dd162aafc4e0b8a206ff3b315d411627274280b248e0821194e598 not found: ID does not exist"
Jul 27 19:01:37.124253 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.124140    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" path="/var/lib/kubelet/pods/6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf/volumes"
Jul 27 19:01:37.345568 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.345538    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-45xpz_4587b5df-0686-4ef4-b0e7-829add7246f4/kube-state-metrics/0.log"
Jul 27 19:01:37.399970 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.399877    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-45xpz_4587b5df-0686-4ef4-b0e7-829add7246f4/kube-rbac-proxy-main/0.log"
Jul 27 19:01:37.448898 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.448869    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-45xpz_4587b5df-0686-4ef4-b0e7-829add7246f4/kube-rbac-proxy-self/0.log"
Jul 27 19:01:37.482348 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.482314    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bd6857454-wqt6s_3f1dd432-d58b-4ea5-91a8-5fd041f6ad54/metrics-server/0.log"
Jul 27 19:01:37.545110 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.545063    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-629cq_1a09cd19-3146-4617-961e-614b1edf601b/node-exporter/0.log"
Jul 27 19:01:37.573132 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.573095    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-629cq_1a09cd19-3146-4617-961e-614b1edf601b/kube-rbac-proxy/0.log"
Jul 27 19:01:37.595029 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.595004    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-629cq_1a09cd19-3146-4617-961e-614b1edf601b/init-textfile/0.log"
Jul 27 19:01:37.869175 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.869141    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4dca4049-bdfd-4125-a942-1ad38c9ea87e/prometheus/0.log"
Jul 27 19:01:37.886416 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.886385    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4dca4049-bdfd-4125-a942-1ad38c9ea87e/config-reloader/0.log"
Jul 27 19:01:37.910007 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.909973    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4dca4049-bdfd-4125-a942-1ad38c9ea87e/thanos-sidecar/0.log"
Jul 27 19:01:37.933576 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.933529    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4dca4049-bdfd-4125-a942-1ad38c9ea87e/kube-rbac-proxy-web/0.log"
Jul 27 19:01:37.954938 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.954910    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4dca4049-bdfd-4125-a942-1ad38c9ea87e/kube-rbac-proxy/0.log"
Jul 27 19:01:37.978654 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:37.978629    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4dca4049-bdfd-4125-a942-1ad38c9ea87e/kube-rbac-proxy-thanos/0.log"
Jul 27 19:01:38.004539 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:38.004503    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4dca4049-bdfd-4125-a942-1ad38c9ea87e/init-config-reloader/0.log"
Jul 27 19:01:38.109984 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:38.109950    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b6f69bf7c-w55bl_57631304-d550-49a7-8e57-fddbecf9e6de/telemeter-client/0.log"
Jul 27 19:01:38.131919 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:38.131821    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b6f69bf7c-w55bl_57631304-d550-49a7-8e57-fddbecf9e6de/reload/0.log"
Jul 27 19:01:38.152730 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:38.152692    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b6f69bf7c-w55bl_57631304-d550-49a7-8e57-fddbecf9e6de/kube-rbac-proxy/0.log"
Jul 27 19:01:39.609123 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:39.609091    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-865bb_e73495df-4bcc-4b05-883e-a539562031b4/networking-console-plugin/0.log"
Jul 27 19:01:40.146343 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:40.146273    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/1.log"
Jul 27 19:01:40.156063 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:40.156023    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2fmlt_e83a9b52-aad4-4ffb-a228-896ef91a710c/console-operator/2.log"
Jul 27 19:01:40.569677 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:40.569644    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f5c7d6b87-6w45f_50e0de24-faf3-4ff5-85e5-7052840da433/console/0.log"
Jul 27 19:01:40.626045 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:40.626011    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-nlscd_ee681144-a38e-4964-9c30-604dfd4e6d71/download-server/0.log"
Jul 27 19:01:41.345686 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.345649    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"]
Jul 27 19:01:41.346008 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.345993    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" containerName="copy"
Jul 27 19:01:41.346069 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.346010    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" containerName="copy"
Jul 27 19:01:41.346069 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.346051    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" containerName="gather"
Jul 27 19:01:41.346069 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.346058    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" containerName="gather"
Jul 27 19:01:41.346206 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.346109    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" containerName="copy"
Jul 27 19:01:41.346206 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.346118    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec1e41c-5ee8-4b4c-bbd7-4a3cac3cbfaf" containerName="gather"
Jul 27 19:01:41.350091 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.350066    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.353091 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.353063    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grcz9\"/\"default-dockercfg-5fzhb\""
Jul 27 19:01:41.353263 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.353063    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grcz9\"/\"openshift-service-ca.crt\""
Jul 27 19:01:41.354048 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.354026    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grcz9\"/\"kube-root-ca.crt\""
Jul 27 19:01:41.357801 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.357473    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"]
Jul 27 19:01:41.410916 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.410870    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jh7m\" (UniqueName: \"kubernetes.io/projected/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-kube-api-access-5jh7m\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.411131 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.410980    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-proc\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.411131 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.411045    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-lib-modules\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.411131 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.411066    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-sys\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.411131 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.411088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-podres\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.511814 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.511774    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-proc\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.512060 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.511834    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-lib-modules\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.512060 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.511856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-sys\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.512060 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.511876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-podres\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.512060 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.511907    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jh7m\" (UniqueName: \"kubernetes.io/projected/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-kube-api-access-5jh7m\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.512060 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.511922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-proc\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.512060 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.511977    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-sys\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.512060 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.512046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-podres\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.512400 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.512045    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-lib-modules\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.520135 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.520096    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jh7m\" (UniqueName: \"kubernetes.io/projected/8a0ec0e3-d237-4b7a-b21c-125e311a3f5a-kube-api-access-5jh7m\") pod \"perf-node-gather-daemonset-jcmrv\" (UID: \"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.662160 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.662033    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:41.804503 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:41.804462    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"]
Jul 27 19:01:41.807146 ip-10-0-129-58 kubenswrapper[2564]: W0727 19:01:41.807112    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a0ec0e3_d237_4b7a_b21c_125e311a3f5a.slice/crio-c50281da3a547d7bd7b0d453582170e096303d9b2dd1c9c2af40ae6fe14b39ab WatchSource:0}: Error finding container c50281da3a547d7bd7b0d453582170e096303d9b2dd1c9c2af40ae6fe14b39ab: Status 404 returned error can't find the container with id c50281da3a547d7bd7b0d453582170e096303d9b2dd1c9c2af40ae6fe14b39ab
Jul 27 19:01:42.089609 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:42.089576    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xt257_332d404e-0e71-485a-8a0d-05951cf4752d/dns/0.log"
Jul 27 19:01:42.110129 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:42.110095    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xt257_332d404e-0e71-485a-8a0d-05951cf4752d/kube-rbac-proxy/0.log"
Jul 27 19:01:42.171570 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:42.171533    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2lkcp_df9a0647-8600-491a-8042-186017147267/dns-node-resolver/0.log"
Jul 27 19:01:42.732238 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:42.732173    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv" event={"ID":"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a","Type":"ContainerStarted","Data":"6d328956b3eadf6516497b21b77cc9b250c1f967abee73ab6a6bde35e2fdd872"}
Jul 27 19:01:42.732716 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:42.732287    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv" event={"ID":"8a0ec0e3-d237-4b7a-b21c-125e311a3f5a","Type":"ContainerStarted","Data":"c50281da3a547d7bd7b0d453582170e096303d9b2dd1c9c2af40ae6fe14b39ab"}
Jul 27 19:01:42.732716 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:42.732313    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:42.748641 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:42.748573    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv" podStartSLOduration=1.748524285 podStartE2EDuration="1.748524285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:01:42.747806476 +0000 UTC m=+5646.200427626" watchObservedRunningTime="2026-07-27 19:01:42.748524285 +0000 UTC m=+5646.201145434"
Jul 27 19:01:42.750683 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:42.750653    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v8rh6_9c64d459-de22-4e66-a489-4c7ac0a62897/node-ca/0.log"
Jul 27 19:01:43.851320 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:43.851282    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mjj85_7cd586f7-6fb6-4d6d-a0fc-2b633a11f5ac/serve-healthcheck-canary/0.log"
Jul 27 19:01:44.314315 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:44.314278    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ptt5w_245f72c6-ba0b-47c0-9631-6966ca016a69/kube-rbac-proxy/0.log"
Jul 27 19:01:44.334192 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:44.334151    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ptt5w_245f72c6-ba0b-47c0-9631-6966ca016a69/exporter/0.log"
Jul 27 19:01:44.355140 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:44.355112    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ptt5w_245f72c6-ba0b-47c0-9631-6966ca016a69/extractor/0.log"
Jul 27 19:01:47.348583 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:47.348540    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bcx6x_d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d/s3-init/0.log"
Jul 27 19:01:47.365423 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:47.365385    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bcx6x_d5fe2eba-eb28-46ea-ba1c-33ce3ab8e78d/download-hf-model/0.log"
Jul 27 19:01:47.398877 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:47.398842    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-wbpcl_74538621-237b-4ae6-bea4-6f4b6127fb93/s3-tls-init-custom/0.log"
Jul 27 19:01:47.414689 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:47.414652    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-wbpcl_74538621-237b-4ae6-bea4-6f4b6127fb93/download-hf-model/0.log"
Jul 27 19:01:47.448790 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:47.448751    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-28gp8_f2b7a693-8130-461c-a5d8-1363948faa60/s3-tls-init-serving/0.log"
Jul 27 19:01:47.464935 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:47.464902    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-28gp8_f2b7a693-8130-461c-a5d8-1363948faa60/download-hf-model/0.log"
Jul 27 19:01:47.496833 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:47.496799    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-d8dz4_32320e1d-7049-4caa-ab3a-f75ee8a8dd1c/seaweedfs/0.log"
Jul 27 19:01:47.546572 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:47.546535    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-49rlc_d89c7371-f884-427b-ace9-a7a4dc4424ab/seaweedfs-tls-serving/0.log"
Jul 27 19:01:48.747525 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:48.747493    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-jcmrv"
Jul 27 19:01:52.949883 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:52.949798    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n252c_95cbf9d1-0fa8-43af-9900-08b4d8cc7051/kube-multus-additional-cni-plugins/0.log"
Jul 27 19:01:52.970558 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:52.970524    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n252c_95cbf9d1-0fa8-43af-9900-08b4d8cc7051/egress-router-binary-copy/0.log"
Jul 27 19:01:52.990219 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:52.990162    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n252c_95cbf9d1-0fa8-43af-9900-08b4d8cc7051/cni-plugins/0.log"
Jul 27 19:01:53.011490 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:53.011457    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n252c_95cbf9d1-0fa8-43af-9900-08b4d8cc7051/bond-cni-plugin/0.log"
Jul 27 19:01:53.036781 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:53.036741    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n252c_95cbf9d1-0fa8-43af-9900-08b4d8cc7051/routeoverride-cni/0.log"
Jul 27 19:01:53.065172 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:53.065141    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n252c_95cbf9d1-0fa8-43af-9900-08b4d8cc7051/whereabouts-cni-bincopy/0.log"
Jul 27 19:01:53.086955 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:53.086918    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n252c_95cbf9d1-0fa8-43af-9900-08b4d8cc7051/whereabouts-cni/0.log"
Jul 27 19:01:53.470287 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:53.470251    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gqmhw_a8024a66-c645-46d6-b1c3-f75dd18abb20/kube-multus/0.log"
Jul 27 19:01:53.564432 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:53.564393    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-flnfk_6b84e87f-4fbf-471c-bf2f-2c364e4d9a82/network-metrics-daemon/0.log"
Jul 27 19:01:53.582704 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:53.582675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-flnfk_6b84e87f-4fbf-471c-bf2f-2c364e4d9a82/kube-rbac-proxy/0.log"
Jul 27 19:01:54.649026 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:54.648951    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-controller/0.log"
Jul 27 19:01:54.667005 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:54.666960    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/0.log"
Jul 27 19:01:54.667217 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:54.667053    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovn-acl-logging/1.log"
Jul 27 19:01:54.687422 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:54.687388    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/kube-rbac-proxy-node/0.log"
Jul 27 19:01:54.709071 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:54.709039    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/kube-rbac-proxy-ovn-metrics/0.log"
Jul 27 19:01:54.726443 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:54.726410    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/northd/0.log"
Jul 27 19:01:54.748920 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:54.748887    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/nbdb/0.log"
Jul 27 19:01:54.768403 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:54.768375    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/sbdb/0.log"
Jul 27 19:01:54.841893 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:54.841853    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fm8gt_b1d3ea62-7e04-4d73-a171-6e573d4e6637/ovnkube-controller/0.log"
Jul 27 19:01:56.001605 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:56.001574    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8wtkw_42f3ab66-e7e3-4565-8357-475969cf2579/network-check-target-container/0.log"
Jul 27 19:01:56.884954 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:56.884923    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-99rcs_a7f67c40-5ff8-4ce1-9ed9-7ac390f9ff5e/iptables-alerter/0.log"
Jul 27 19:01:57.541894 ip-10-0-129-58 kubenswrapper[2564]: I0727 19:01:57.541854    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-26rmm_08be5885-b32c-4799-8157-d0c0000bf3ae/tuned/0.log"