Jul 06 20:33:23.936875 ip-10-0-134-39 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 06 20:33:23.936887 ip-10-0-134-39 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 06 20:33:23.936895 ip-10-0-134-39 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 06 20:33:23.937110 ip-10-0-134-39 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 06 20:33:34.045626 ip-10-0-134-39 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 06 20:33:34.045646 ip-10-0-134-39 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot fbeb159ada504dd7af009b80b8cbd666 --
Jul 06 20:35:24.558336 ip-10-0-134-39 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 20:35:24.878193 ip-10-0-134-39 kubenswrapper[2541]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:35:24.878193 ip-10-0-134-39 kubenswrapper[2541]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 20:35:24.878193 ip-10-0-134-39 kubenswrapper[2541]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:35:24.878193 ip-10-0-134-39 kubenswrapper[2541]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 20:35:24.878193 ip-10-0-134-39 kubenswrapper[2541]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 20:35:24.880165 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.880078    2541 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 20:35:24.884721 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884704    2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:24.884721 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884719    2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:24.884721 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884723    2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884726    2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884729    2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884731    2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884734    2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884737    2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884742    2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884746    2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884749    2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884752    2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884755    2541 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884757    2541 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884760    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884763    2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884765    2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884768    2541 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884771    2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884773    2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884781    2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:24.884815 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884784    2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884786    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884789    2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884791    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884794    2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884796    2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884799    2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884801    2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884804    2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884806    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884809    2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884811    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884813    2541 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884816    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884819    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884823    2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884827    2541 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884830    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884833    2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884836    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:24.885302 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884839    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884841    2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884843    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884846    2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884848    2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884851    2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884853    2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884856    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884858    2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884861    2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884863    2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884867    2541 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884876    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884880    2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884883    2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884886    2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884889    2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884892    2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884895    2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:24.886026 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884898    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884900    2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884903    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884906    2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884908    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884911    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884913    2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884916    2541 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884919    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884921    2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884924    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884927    2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884929    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884932    2541 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884934    2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884937    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884939    2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884941    2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884944    2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884946    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:24.886495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884949    2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884951    2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884954    2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884956    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884959    2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884961    2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884964    2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884966    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884970    2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884973    2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884975    2541 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884978    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884980    2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884983    2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884985    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884987    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884990    2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884992    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:24.886979 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.884995    2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:24.888702 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888686    2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:24.888702 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888702    2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888707    2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888710    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888713    2541 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888716    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888719    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888722    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888724    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888727    2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888729    2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888732    2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888734    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888737    2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888739    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888742    2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888744    2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888746    2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888749    2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888751    2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888754    2541 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:24.888764 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888756    2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888758    2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888760    2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888763    2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888765    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888768    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888772    2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888774    2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888777    2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888780    2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888782    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888784    2541 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888787    2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888789    2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888792    2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888794    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888797    2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888799    2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888802    2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:24.889300 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888804    2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888807    2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888809    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888811    2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888814    2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888816    2541 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888818    2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888821    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888823    2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888825    2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888828    2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888830    2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888833    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888836    2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888838    2541 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888842    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888844    2541 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888847    2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888849    2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888852    2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:24.889789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888855    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888857    2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888860    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888862    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888866    2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888869    2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888872    2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888875    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888878    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888881    2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888883    2541 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888886    2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888889    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888892    2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888895    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888897    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888900    2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888902    2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888905    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888908    2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:24.890312 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888910    2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888912    2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888915    2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888919    2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888921    2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888924    2541 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888926    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888929    2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888931    2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888934    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888939    2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888942    2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888945    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888947    2541 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888950    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888952    2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888955    2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888958    2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.888962    2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:24.890778 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889342    2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889347    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889351    2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889353    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889356    2541 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889360    2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889363    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889365    2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889368    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889371    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889374    2541 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889377    2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889379    2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889382    2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889385    2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889387    2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889391    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889393    2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889396    2541 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889399    2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:24.891243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889402    2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889405    2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889407    2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889410    2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889412    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889415    2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889417    2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889420    2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889422    2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889425    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889427    2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889429    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889432    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889434    2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889437    2541 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889439    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889441    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889444    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889447    2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889449    2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:24.891716 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889452    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889454    2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889457    2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889459    2541 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889462    2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889464    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889467    2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889477    2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889480    2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889483    2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889486    2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889489    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889492    2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889497    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889499    2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889502    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889504    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889507    2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889509    2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:24.892199 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889512    2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889514    2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889517    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889519    2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889522    2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889524    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889526    2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889529    2541 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889531    2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889534    2541 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889550    2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889553    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889555    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889558    2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889560    2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889562    2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889564    2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889567    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889569    2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:24.892683 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889571    2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889574    2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889576    2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889578    2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889581    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889589    2541 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889592    2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889595    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889597    2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889600    2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889602    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889606    2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889609    2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889612    2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889615    2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889618    2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889620    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889623    2541 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889626    2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889628    2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:24.893180 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.889631    2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889698    2541 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889705    2541 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889711    2541 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889715    2541 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889719    2541 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889721    2541 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889725    2541 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889729    2541 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889732    2541 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889735    2541 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889738    2541 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889741    2541 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889744    2541 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889747    2541 flags.go:64] FLAG: --cgroup-root=""
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889749    2541 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889752    2541 flags.go:64] FLAG: --client-ca-file=""
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889755    2541 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889758    2541 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889762    2541 flags.go:64] FLAG: --cluster-domain=""
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889765    2541 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889768    2541 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889770    2541 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889774    2541 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 20:35:24.893662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889777    2541 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889782    2541 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889785    2541 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889788    2541 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889791    2541 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889794    2541 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889796    2541 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889799    2541 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889803    2541 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889807    2541 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889810    2541 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889812    2541 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889815    2541 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889818    2541 flags.go:64] FLAG: --enable-server="true"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889820    2541 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889825    2541 flags.go:64] FLAG: --event-burst="100"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889827    2541 flags.go:64] FLAG: --event-qps="50"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889830    2541 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889833    2541 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889836    2541 flags.go:64] FLAG: --eviction-hard=""
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889840    2541 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889842    2541 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889845    2541 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889847    2541 flags.go:64] FLAG: --eviction-soft=""
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889850    2541 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 20:35:24.894235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889853    2541 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889856    2541 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889859    2541 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889862    2541 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889865    2541 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889868    2541 flags.go:64] FLAG: --feature-gates=""
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889871    2541 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889874    2541 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889877    2541 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889880    2541 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889883    2541 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889886    2541 flags.go:64] FLAG: --help="false"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889888    2541 flags.go:64] FLAG: --hostname-override="ip-10-0-134-39.ec2.internal"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889891    2541 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889893    2541 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889896    2541 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889899    2541 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889902    2541 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889905    2541 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889908    2541 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889910    2541 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889913    2541 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889915    2541 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889918    2541 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 20:35:24.894854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889921    2541 flags.go:64] FLAG: --kube-reserved=""
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889923    2541 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889926    2541 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889929    2541 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889931    2541 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889934    2541 flags.go:64] FLAG: --lock-file=""
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889936    2541 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889939    2541 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889941    2541 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889946    2541 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889950    2541 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889953    2541 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889955    2541 flags.go:64] FLAG: --logging-format="text"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889958    2541 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889961    2541 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889964    2541 flags.go:64] FLAG: --manifest-url=""
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889966    2541 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889971    2541 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889973    2541 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889977    2541 flags.go:64] FLAG: --max-pods="110"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889980    2541 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889983    2541 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889985    2541 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889988    2541 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889990    2541 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 20:35:24.895433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889993    2541 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.889996    2541 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890002    2541 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890005    2541 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890008    2541 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890010    2541 flags.go:64] FLAG: --pod-cidr=""
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890013    2541 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890018    2541 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890021    2541 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890023    2541 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890026    2541 flags.go:64] FLAG: --port="10250"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890029    2541 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890031    2541 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a2e0df92f8ca830b"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890034    2541 flags.go:64] FLAG: --qos-reserved=""
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890037    2541 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890040    2541 flags.go:64] FLAG: --register-node="true"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890056    2541 flags.go:64] FLAG: --register-with-taints=""
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890060    2541 flags.go:64] FLAG: --registry-burst="10"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890063    2541 flags.go:64] FLAG: --registry-qps="5"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890065    2541 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890068    2541 flags.go:64] FLAG: --reserved-memory=""
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890071    2541 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890074    2541 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890077    2541 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890079    2541 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 20:35:24.896027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890082    2541 flags.go:64] FLAG: --runonce="false"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890084    2541 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890087    2541 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890090    2541 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890093    2541 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890095    2541 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890098    2541 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890101    2541 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890103    2541 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890106    2541 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890109    2541 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890111    2541 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890114    2541 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890116    2541 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890119    2541 flags.go:64] FLAG: --system-cgroups=""
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890121    2541 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890128    2541 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890131    2541 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890133    2541 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890137    2541 flags.go:64] FLAG: --tls-min-version=""
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890140    2541 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890142    2541 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890145    2541 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890147    2541 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890150    2541 flags.go:64] FLAG: --v="2"
Jul 06 20:35:24.896665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890154    2541 flags.go:64] FLAG: --version="false"
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890158    2541 flags.go:64] FLAG: --vmodule=""
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890162    2541 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.890165    2541 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890276    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890279    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890282    2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890285    2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890287    2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890290    2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890293    2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890295    2541 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890298    2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890301    2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890303    2541 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890306    2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890308    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890311    2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890314    2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890316    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:24.897318 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890319    2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890322    2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890324    2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890329    2541 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890331    2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890334    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890336    2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890338    2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890341    2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890343    2541 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890346    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890348    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890351    2541 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890353    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890356    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890358    2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890364    2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890368    2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890371    2541 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890373    2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:24.897803 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890375    2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890378    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890380    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890383    2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890385    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890388    2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890390    2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890393    2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890395    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890397    2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890400    2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890402    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890405    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890407    2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890410    2541 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890413    2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890415    2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890418    2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890420    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:24.898303 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890422    2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890424    2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890427    2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890430    2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890432    2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890435    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890437    2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890439    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890442    2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890446    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890448    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890451    2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890453    2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890455    2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890458    2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890460    2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890462    2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890465    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890467    2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890469    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:24.898756 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890472    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890474    2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890476    2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890478    2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890481    2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890483    2541 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890486    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890488    2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890491    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890494    2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890496    2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890498    2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890502    2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890505    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890508    2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890511    2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890514    2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890516    2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890519    2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:24.899266 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890522    2541 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890524    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890532    2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890535    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.890537    2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.891082    2541 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.897458    2541 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.897475    2541 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.898098    2541 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.898105    2541 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898163    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898167    2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898170    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898174    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898176    2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:24.899789 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898179    2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898182    2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898185    2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898187    2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898190    2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898192    2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898195    2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898198    2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898200    2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898202    2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898205    2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898207    2541 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898210    2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898213    2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898216    2541 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898218    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898220    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898223    2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898225    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:24.900234 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898228    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898230    2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898233    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898235    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898238    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898241    2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898244    2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898246    2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898248    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898251    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898253    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898256    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898258    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898260    2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898262    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898265    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898267    2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898270    2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898272    2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898274    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:24.900703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898276    2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898278    2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898281    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898283    2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898285    2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898288    2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898290    2541 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898293    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898295    2541 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898297    2541 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898299    2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898301    2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898304    2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898306    2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898309    2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898312    2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898315    2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898318    2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898320    2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898323    2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:24.901213 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898325    2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898328    2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898330    2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898332    2541 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898335    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898337    2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898340    2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898342    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898344    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898346    2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898349    2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898351    2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898353    2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898355    2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898358    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898362    2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898365    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898368    2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898370    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:24.901703 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898374    2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898377    2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898380    2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898382    2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898385    2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898388    2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898390    2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898393    2541 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898395    2541 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898398    2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898400    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898403    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898406    2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898408    2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898412    2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:24.902161 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898415    2541 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.898419    2541 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898544    2541 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898549    2541 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898552    2541 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898554    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898556    2541 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898559    2541 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898561    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898563    2541 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898565    2541 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898568    2541 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898570    2541 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898572    2541 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898575    2541 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 20:35:24.902528 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898577    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898579    2541 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898581    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898583    2541 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898586    2541 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898588    2541 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898591    2541 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898594    2541 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898597    2541 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898600    2541 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898603    2541 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898605    2541 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898608    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898611    2541 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898613    2541 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898616    2541 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898618    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898621    2541 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898623    2541 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898625    2541 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 20:35:24.902908 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898627    2541 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898630    2541 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898632    2541 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898634    2541 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898636    2541 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898639    2541 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898641    2541 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898644    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898646    2541 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898648    2541 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898651    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898653    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898655    2541 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898657    2541 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898660    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898662    2541 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898664    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898668    2541 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898671    2541 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 20:35:24.903389 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898673    2541 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898676    2541 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898678    2541 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898681    2541 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898684    2541 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898686    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898689    2541 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898691    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898694    2541 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898696    2541 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898699    2541 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898701    2541 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898703    2541 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898705    2541 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898708    2541 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898710    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898712    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898715    2541 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898717    2541 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898720    2541 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 20:35:24.903842 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898723    2541 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898725    2541 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898728    2541 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898730    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898732    2541 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898735    2541 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898737    2541 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898739    2541 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898741    2541 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898744    2541 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898746    2541 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898748    2541 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898750    2541 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898753    2541 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898755    2541 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898757    2541 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898760    2541 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898762    2541 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898765    2541 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898768    2541 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 20:35:24.904390 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898770    2541 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898772    2541 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898775    2541 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898778    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898780    2541 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898782    2541 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:24.898784    2541 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.898788    2541 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.898887    2541 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.901213    2541 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.902414    2541 server.go:1019] "Starting client certificate rotation"
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.902510    2541 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.902543    2541 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 20:35:24.904930 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.904515    2541 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:35:24.906074 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.906059    2541 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 20:35:24.914886 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.914865    2541 log.go:25] "Validated CRI v1 runtime API"
Jul 06 20:35:24.916830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.916812    2541 log.go:25] "Validated CRI v1 image API"
Jul 06 20:35:24.920126 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.920110    2541 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 20:35:24.922260 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.922242    2541 fs.go:135] Filesystem UUIDs: map[40735e2a-cbde-477d-9ce2-a0fcdf5dd6d5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b3fe641a-bfa8-439d-b104-639b8335fbd5:/dev/nvme0n1p4]
Jul 06 20:35:24.922310 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.922260    2541 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 20:35:24.926974 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.926869    2541 manager.go:216] Machine: {Timestamp:2026-07-06 20:35:24.925768221 +0000 UTC m=+0.267670928 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3186495 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b17dcaf7e98274840161286666835 SystemUUID:ec2b17dc-af7e-9827-4840-161286666835 BootID:fbeb159a-da50-4dd7-af00-9b80b8cbd666 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c4:b1:b1:0d:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c4:b1:b1:0d:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:8b:e4:ee:34:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 20:35:24.926974 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.926968    2541 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 20:35:24.927093 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.927039    2541 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 20:35:24.929648 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.929625    2541 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 20:35:24.929796 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.929651    2541 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-39.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 20:35:24.930312 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.930302    2541 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 20:35:24.930345 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.930315    2541 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 20:35:24.930345 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.930326    2541 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:35:24.931035 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.931023    2541 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 20:35:24.932271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.932261    2541 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 20:35:24.934237 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.934220    2541 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:35:24.936145 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.936133    2541 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:35:24.936249 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.936240    2541 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 20:35:24.936277 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.936267    2541 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 20:35:24.936303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.936277    2541 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 20:35:24.936303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.936290    2541 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 20:35:24.936303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.936298    2541 kubelet.go:398] "Adding apiserver pod source"
Jul 06 20:35:24.936423 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.936305    2541 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 20:35:24.936721 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.936702    2541 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:35:24.936721 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.936728    2541 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 20:35:24.938489 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.938473    2541 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 20:35:24.940290 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.940277    2541 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 20:35:24.940343 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.940306    2541 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 20:35:24.941271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941259    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 20:35:24.941335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941275    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 20:35:24.941335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941280    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 20:35:24.941335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941286    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 20:35:24.941335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941295    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 20:35:24.941335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941301    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 20:35:24.941335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941309    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 20:35:24.941335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941315    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 20:35:24.941335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941322    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 20:35:24.941335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941327    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 20:35:24.941570 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941340    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 20:35:24.941570 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.941345    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 20:35:24.942215 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.942205    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 20:35:24.942255 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.942216    2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 20:35:24.945498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.945481    2541 server.go:1268] "Started kubelet"
Jul 06 20:35:24.945637 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.945586    2541 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 20:35:24.945721 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.945656    2541 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 20:35:24.945721 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.945668    2541 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 20:35:24.946472 ip-10-0-134-39 systemd[1]: Started Kubernetes Kubelet.
Jul 06 20:35:24.947061 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.946971    2541 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 20:35:24.950230 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.950204    2541 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-39.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 20:35:24.950373 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:24.950349    2541 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-39.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 20:35:24.950497 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:24.950472    2541 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 20:35:24.950700 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.950684    2541 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 20:35:24.951224 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:24.951195    2541 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 20:35:24.952505 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.952481    2541 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7ctfp"
Jul 06 20:35:24.953342 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:24.952597    2541 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-39.ec2.internal.18bfcdae106af5f9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-39.ec2.internal,UID:ip-10-0-134-39.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-39.ec2.internal,},FirstTimestamp:2026-07-06 20:35:24.945454585 +0000 UTC m=+0.287357295,LastTimestamp:2026-07-06 20:35:24.945454585 +0000 UTC m=+0.287357295,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-39.ec2.internal,}"
Jul 06 20:35:24.955777 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.955761    2541 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 20:35:24.955777 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.955766    2541 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 20:35:24.955881 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:24.955843    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:24.956319 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956301    2541 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 20:35:24.956411 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956390    2541 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 20:35:24.956411 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956402    2541 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 20:35:24.956545 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956525    2541 factory.go:55] Registering systemd factory
Jul 06 20:35:24.956545 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956539    2541 factory.go:223] Registration of the systemd container factory successfully
Jul 06 20:35:24.956632 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956555    2541 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 20:35:24.956632 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956567    2541 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 20:35:24.956771 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956756    2541 factory.go:153] Registering CRI-O factory
Jul 06 20:35:24.956771 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956772    2541 factory.go:223] Registration of the crio container factory successfully
Jul 06 20:35:24.956902 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956821    2541 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 20:35:24.956902 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956846    2541 factory.go:103] Registering Raw factory
Jul 06 20:35:24.956902 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.956860    2541 manager.go:1196] Started watching for new ooms in manager
Jul 06 20:35:24.957445 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.957431    2541 manager.go:349] Starting recovery of all containers
Jul 06 20:35:24.963076 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.963057    2541 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7ctfp"
Jul 06 20:35:24.963657 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:24.963630    2541 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 06 20:35:24.963783 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:24.963760    2541 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-39.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 06 20:35:24.968187 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.967998    2541 manager.go:354] Recovery completed
Jul 06 20:35:24.974291 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.974280    2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:24.976370 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.976348    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:24.976470 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.976387    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:24.976470 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.976411    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:24.977566 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.977552    2541 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 20:35:24.977566 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.977565    2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:24.977676 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.977564    2541 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 20:35:24.977676 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.977636    2541 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 20:35:24.978408 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.978392    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:24.978491 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.978445    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:24.978491 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.978461    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:24.979601 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.979589    2541 policy_none.go:49] "None policy: Start"
Jul 06 20:35:24.979641 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.979605    2541 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 20:35:24.979641 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.979615    2541 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 20:35:24.981550 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:24.981536    2541 policy_none.go:47] "Start"
Jul 06 20:35:25.029661 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.029643    2541 manager.go:337] "Starting Device Plugin manager"
Jul 06 20:35:25.029760 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.029692    2541 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 20:35:25.029760 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.029706    2541 server.go:87] "Starting device plugin registration server"
Jul 06 20:35:25.029979 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.029966    2541 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 20:35:25.030039 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.029980    2541 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 20:35:25.030101 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.030078    2541 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 20:35:25.030184 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.030170    2541 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 20:35:25.030242 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.030185    2541 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 20:35:25.030673 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.030655    2541 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 20:35:25.030742 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.030707    2541 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.085670 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.085638    2541 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 20:35:25.087795 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.086845    2541 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 20:35:25.087795 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.086886    2541 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 20:35:25.087795 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.086906    2541 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 20:35:25.087795 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.086936    2541 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 20:35:25.089718 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.089700    2541 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:25.131022 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.130962    2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:25.132365 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.132344    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:25.132488 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.132375    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:25.132488 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.132386    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:25.132488 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.132420    2541 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.147241 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.147217    2541 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.147241 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.147245    2541 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-39.ec2.internal\": node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.161625 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.161598    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.187019 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.186994    2541 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal"]
Jul 06 20:35:25.187112 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.187075    2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:25.188486 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.188472    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:25.188536 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.188503    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:25.188536 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.188516    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:25.190870 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.190858    2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:25.191011 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.191000    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.191059 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.191024    2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:25.191548 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.191531    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:25.191548 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.191543    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:25.191699 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.191555    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:25.191699 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.191565    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:25.191699 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.191566    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:25.191699 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.191660    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:25.193779 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.193765    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.193846 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.193787    2541 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 20:35:25.194422 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.194405    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 20:35:25.194526 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.194430    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 20:35:25.194526 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.194442    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeHasSufficientPID"
Jul 06 20:35:25.215564 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.215542    2541 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-39.ec2.internal\" not found" node="ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.219414 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.219398    2541 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-39.ec2.internal\" not found" node="ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.258016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.257994    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcc7400c82ada97bc7d9b4033908bae2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal\" (UID: \"bcc7400c82ada97bc7d9b4033908bae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.258136 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.258019    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc7400c82ada97bc7d9b4033908bae2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal\" (UID: \"bcc7400c82ada97bc7d9b4033908bae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.258136 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.258035    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/487ce3ac867cb6a29534fc53f0fccb26-config\") pod \"kube-apiserver-proxy-ip-10-0-134-39.ec2.internal\" (UID: \"487ce3ac867cb6a29534fc53f0fccb26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.262102 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.262090    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.358483 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.358452    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcc7400c82ada97bc7d9b4033908bae2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal\" (UID: \"bcc7400c82ada97bc7d9b4033908bae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.358483 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.358486    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc7400c82ada97bc7d9b4033908bae2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal\" (UID: \"bcc7400c82ada97bc7d9b4033908bae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.358641 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.358501    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/487ce3ac867cb6a29534fc53f0fccb26-config\") pod \"kube-apiserver-proxy-ip-10-0-134-39.ec2.internal\" (UID: \"487ce3ac867cb6a29534fc53f0fccb26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.358641 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.358544    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/487ce3ac867cb6a29534fc53f0fccb26-config\") pod \"kube-apiserver-proxy-ip-10-0-134-39.ec2.internal\" (UID: \"487ce3ac867cb6a29534fc53f0fccb26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.358641 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.358555    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcc7400c82ada97bc7d9b4033908bae2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal\" (UID: \"bcc7400c82ada97bc7d9b4033908bae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.358641 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.358554    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcc7400c82ada97bc7d9b4033908bae2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal\" (UID: \"bcc7400c82ada97bc7d9b4033908bae2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.362553 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.362538    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.463446 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.463376    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.518571 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.518536    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.521983 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.521961    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:25.564511 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.564483    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.665041 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.664996    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.765671 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.765612    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.866135 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.866109    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.902296 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.902265    2541 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 20:35:25.902819 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.902402    2541 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 20:35:25.966376 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:25.966349    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:25.967439 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.967418    2541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:30:25 +0000 UTC" deadline="2028-03-24 02:54:16.576579858 +0000 UTC"
Jul 06 20:35:25.967439 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:25.967438    2541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15030h18m50.609144563s"
Jul 06 20:35:26.067443 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:26.067371    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:26.095677 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:26.095649    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487ce3ac867cb6a29534fc53f0fccb26.slice/crio-424330a25388d43c0d0a1ce2719046896dacabaaa2b41b02fe2f53dfad56da71 WatchSource:0}: Error finding container 424330a25388d43c0d0a1ce2719046896dacabaaa2b41b02fe2f53dfad56da71: Status 404 returned error can't find the container with id 424330a25388d43c0d0a1ce2719046896dacabaaa2b41b02fe2f53dfad56da71
Jul 06 20:35:26.096065 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:26.096030    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc7400c82ada97bc7d9b4033908bae2.slice/crio-96f36bf60cefa259e3a23354e9763af29520f7b4eef25100146a0675e0f55da3 WatchSource:0}: Error finding container 96f36bf60cefa259e3a23354e9763af29520f7b4eef25100146a0675e0f55da3: Status 404 returned error can't find the container with id 96f36bf60cefa259e3a23354e9763af29520f7b4eef25100146a0675e0f55da3
Jul 06 20:35:26.100229 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.100211    2541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:35:26.168321 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:26.168288    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:26.268837 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:26.268803    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:26.369359 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:26.369285    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:26.433205 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.433175    2541 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:26.469496 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:26.469469    2541 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-39.ec2.internal\" not found"
Jul 06 20:35:26.498988 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.498961    2541 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:26.515214 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.514598    2541 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:26.556179 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.556154    2541 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:26.568643 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.568614    2541 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:35:26.569624 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.569605    2541 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal"
Jul 06 20:35:26.577456 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.577433    2541 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 20:35:26.937228 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.937190    2541 apiserver.go:52] "Watching apiserver"
Jul 06 20:35:26.943022 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.942996    2541 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 20:35:26.952777 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.952266    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-r89gp","openshift-dns/node-resolver-784f2","openshift-image-registry/node-ca-vz5nr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal","openshift-multus/multus-additional-cni-plugins-t7wgp","openshift-multus/network-metrics-daemon-4zbdm","kube-system/global-pull-secret-syncer-p68f6","kube-system/konnectivity-agent-466g2","openshift-multus/multus-fh5b5","openshift-network-diagnostics/network-check-target-2vksd","openshift-network-operator/iptables-alerter-w4qb2","openshift-ovn-kubernetes/ovnkube-node-gpn7x","kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"]
Jul 06 20:35:26.955095 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.955073    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:26.956227 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.956209    2541 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 20:35:26.957262 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.957241    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:26.957729 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.957708    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 20:35:26.957825 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.957729    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:26.957825 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.957776    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ch9rz\""
Jul 06 20:35:26.958584 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.958562    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:26.958994 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.958977    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 20:35:26.959405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.959387    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 20:35:26.959704 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.959686    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kk25g\""
Jul 06 20:35:26.960055 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.960036    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:26.962404 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.962351    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 20:35:26.962692 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.962673    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 20:35:26.962809 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.962789    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 20:35:26.963029 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.963007    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rhgxp\""
Jul 06 20:35:26.963275 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.963256    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:26.964992 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.964972    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 20:35:26.965416 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.965399    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-km7t8\""
Jul 06 20:35:26.965635 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.965621    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 20:35:26.965865 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.965850    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 20:35:26.966078 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.966061    2541 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 20:35:26.966660 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.966634    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:26.966778 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:26.966752    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:26.967819 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.967795    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/821107a9-9583-4e63-869d-bff90d64af15-iptables-alerter-script\") pod \"iptables-alerter-w4qb2\" (UID: \"821107a9-9583-4e63-869d-bff90d64af15\") " pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:26.967917 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.967843    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/821107a9-9583-4e63-869d-bff90d64af15-host-slash\") pod \"iptables-alerter-w4qb2\" (UID: \"821107a9-9583-4e63-869d-bff90d64af15\") " pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:26.967917 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.967869    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hhv\" (UniqueName: \"kubernetes.io/projected/821107a9-9583-4e63-869d-bff90d64af15-kube-api-access-24hhv\") pod \"iptables-alerter-w4qb2\" (UID: \"821107a9-9583-4e63-869d-bff90d64af15\") " pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:26.967917 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.967893    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73332827-2d68-4a90-ba2c-d3ed85704330-hosts-file\") pod \"node-resolver-784f2\" (UID: \"73332827-2d68-4a90-ba2c-d3ed85704330\") " pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:26.967917 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.967915    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73332827-2d68-4a90-ba2c-d3ed85704330-tmp-dir\") pod \"node-resolver-784f2\" (UID: \"73332827-2d68-4a90-ba2c-d3ed85704330\") " pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:26.968128 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.967942    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jwl6\" (UniqueName: \"kubernetes.io/projected/73332827-2d68-4a90-ba2c-d3ed85704330-kube-api-access-5jwl6\") pod \"node-resolver-784f2\" (UID: \"73332827-2d68-4a90-ba2c-d3ed85704330\") " pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:26.969126 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.969104    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:26.969334 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:26.969289    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:26.971524 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.971504    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:26.971616 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:26.971580    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:26.974002 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.973971    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:26.976341 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.976178    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 20:35:26.976341 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.976218    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v8mrf\""
Jul 06 20:35:26.976341 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.976219    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 20:35:26.976768 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.976749    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:26.978976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.978954    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 20:35:26.978976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.978974    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qvfkj\""
Jul 06 20:35:26.979150 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.979116    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 20:35:26.979243 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.979211    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 20:35:26.979243 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.979228    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 20:35:26.979913 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.979892    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:26.981979 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.981963    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x9zlt\""
Jul 06 20:35:26.982164 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.981982    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:35:26.982405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.982385    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:26.982498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.982412    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 20:35:26.984984 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.984966    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 20:35:26.985206 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.985192    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rsgp7\""
Jul 06 20:35:26.985451 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.985427    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 20:35:26.985613 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.985597    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:26.987581 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.987432    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 20:35:26.987581 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.987504    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c9vwp\""
Jul 06 20:35:26.987756 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.987623    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 20:35:26.987796 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.987762    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 20:35:26.987874 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.987857    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 20:35:26.988310 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.988227    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 20:35:26.988488 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.988470    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 20:35:26.989906 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.989887    2541 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zskm6"
Jul 06 20:35:26.999095 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:26.999074    2541 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zskm6"
Jul 06 20:35:27.034083 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.034043    2541 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 20:35:27.057891 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.057865    2541 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 20:35:27.068222 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068198    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-system-cni-dir\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.068323 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068238    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-var-lib-cni-multus\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.068323 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068266    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.068433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068313    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-run-ovn\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.068433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068365    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.068433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068421    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/821107a9-9583-4e63-869d-bff90d64af15-iptables-alerter-script\") pod \"iptables-alerter-w4qb2\" (UID: \"821107a9-9583-4e63-869d-bff90d64af15\") " pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:27.068585 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068471    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-socket-dir-parent\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.068585 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068509    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:27.068585 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068537    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.068585 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068568    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40b3a754-3e30-4941-8978-ffe85a952f3a-ovn-node-metrics-cert\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.068750 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068598    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-daemon-config\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.068750 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068617    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-sysconfig\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.068750 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068639    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-sys\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.068750 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068655    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppsp\" (UniqueName: \"kubernetes.io/projected/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-kube-api-access-dppsp\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.068750 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068672    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/448af9b4-632e-4e75-991a-f1a5138b9a52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.068750 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068690    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-kubelet\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.068750 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068728    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7mw2\" (UniqueName: \"kubernetes.io/projected/4f3513df-fc03-42e5-86fc-bedae9f21fc0-kube-api-access-z7mw2\") pod \"node-ca-vz5nr\" (UID: \"4f3513df-fc03-42e5-86fc-bedae9f21fc0\") " pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068798    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068841    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktk8\" (UniqueName: \"kubernetes.io/projected/9f19dafe-c8e6-4053-9d18-542a67692a26-kube-api-access-wktk8\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068871    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f19dafe-c8e6-4053-9d18-542a67692a26-cni-binary-copy\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068898    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-hostroot\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068923    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-lib-modules\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068947    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-log-socket\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068972    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.068998    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6njq\" (UniqueName: \"kubernetes.io/projected/1b599d48-d632-4337-9961-be63b99793e6-kube-api-access-w6njq\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069037    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12201c1c-c44f-4ce1-8d69-228ae73bd969-kubelet-config\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069060    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-tuned\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069113    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-tmp\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069142    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-var-lib-openvswitch\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069166    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-cni-netd\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069071    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/821107a9-9583-4e63-869d-bff90d64af15-iptables-alerter-script\") pod \"iptables-alerter-w4qb2\" (UID: \"821107a9-9583-4e63-869d-bff90d64af15\") " pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:27.069193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069192    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40b3a754-3e30-4941-8978-ffe85a952f3a-env-overrides\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069240    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-registration-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069282    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-device-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069326    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-var-lib-cni-bin\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069347    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr8x7\" (UniqueName: \"kubernetes.io/projected/4df480a7-e4bb-46d9-87d6-e07815d4f87b-kube-api-access-wr8x7\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069367    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-host\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069404    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-cnibin\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069458    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73332827-2d68-4a90-ba2c-d3ed85704330-hosts-file\") pod \"node-resolver-784f2\" (UID: \"73332827-2d68-4a90-ba2c-d3ed85704330\") " pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069500    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-run-k8s-cni-cncf-io\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069527    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-conf-dir\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069556    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-run-multus-certs\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069578    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73332827-2d68-4a90-ba2c-d3ed85704330-hosts-file\") pod \"node-resolver-784f2\" (UID: \"73332827-2d68-4a90-ba2c-d3ed85704330\") " pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069586    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-sysctl-d\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069609    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-run-systemd\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069637    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-etc-openvswitch\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069664    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-var-lib-kubelet\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069687    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:27.069802 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069721    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-var-lib-kubelet\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069751    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/448af9b4-632e-4e75-991a-f1a5138b9a52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069785    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-systemd-units\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069813    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f3513df-fc03-42e5-86fc-bedae9f21fc0-serviceca\") pod \"node-ca-vz5nr\" (UID: \"4f3513df-fc03-42e5-86fc-bedae9f21fc0\") " pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069838    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069868    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/821107a9-9583-4e63-869d-bff90d64af15-host-slash\") pod \"iptables-alerter-w4qb2\" (UID: \"821107a9-9583-4e63-869d-bff90d64af15\") " pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069906    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-run-netns\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069941    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/821107a9-9583-4e63-869d-bff90d64af15-host-slash\") pod \"iptables-alerter-w4qb2\" (UID: \"821107a9-9583-4e63-869d-bff90d64af15\") " pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.069992    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-system-cni-dir\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070017    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-sys-fs\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070066    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-cnibin\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070092    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-modprobe-d\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070115    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-sysctl-conf\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070155    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/448af9b4-632e-4e75-991a-f1a5138b9a52-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070193    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8cd\" (UniqueName: \"kubernetes.io/projected/448af9b4-632e-4e75-991a-f1a5138b9a52-kube-api-access-2h8cd\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070218    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59-konnectivity-ca\") pod \"konnectivity-agent-466g2\" (UID: \"2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59\") " pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:27.070605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070246    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73332827-2d68-4a90-ba2c-d3ed85704330-tmp-dir\") pod \"node-resolver-784f2\" (UID: \"73332827-2d68-4a90-ba2c-d3ed85704330\") " pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070269    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-run\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070293    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-cni-bin\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070359    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40b3a754-3e30-4941-8978-ffe85a952f3a-ovnkube-config\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070394    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40b3a754-3e30-4941-8978-ffe85a952f3a-ovnkube-script-lib\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070418    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f3513df-fc03-42e5-86fc-bedae9f21fc0-host\") pod \"node-ca-vz5nr\" (UID: \"4f3513df-fc03-42e5-86fc-bedae9f21fc0\") " pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070445    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-kubernetes\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070470    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-os-release\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070493    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-run-netns\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070542    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-cni-dir\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070567    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73332827-2d68-4a90-ba2c-d3ed85704330-tmp-dir\") pod \"node-resolver-784f2\" (UID: \"73332827-2d68-4a90-ba2c-d3ed85704330\") " pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070571    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hhv\" (UniqueName: \"kubernetes.io/projected/821107a9-9583-4e63-869d-bff90d64af15-kube-api-access-24hhv\") pod \"iptables-alerter-w4qb2\" (UID: \"821107a9-9583-4e63-869d-bff90d64af15\") " pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070601    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jwl6\" (UniqueName: \"kubernetes.io/projected/73332827-2d68-4a90-ba2c-d3ed85704330-kube-api-access-5jwl6\") pod \"node-resolver-784f2\" (UID: \"73332827-2d68-4a90-ba2c-d3ed85704330\") " pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070638    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-etc-kubernetes\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070665    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-systemd\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070693    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-socket-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070715    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59-agent-certs\") pod \"konnectivity-agent-466g2\" (UID: \"2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59\") " pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:27.071361 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070741    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-os-release\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.071931 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070765    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-slash\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.071931 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070797    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-node-log\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.071931 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070823    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b876p\" (UniqueName: \"kubernetes.io/projected/40b3a754-3e30-4941-8978-ffe85a952f3a-kube-api-access-b876p\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.071931 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070845    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12201c1c-c44f-4ce1-8d69-228ae73bd969-dbus\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:27.071931 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.070867    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-run-openvswitch\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.075981 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.075960    2541 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 20:35:27.079624 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.079601    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jwl6\" (UniqueName: \"kubernetes.io/projected/73332827-2d68-4a90-ba2c-d3ed85704330-kube-api-access-5jwl6\") pod \"node-resolver-784f2\" (UID: \"73332827-2d68-4a90-ba2c-d3ed85704330\") " pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:27.079624 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.079619    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hhv\" (UniqueName: \"kubernetes.io/projected/821107a9-9583-4e63-869d-bff90d64af15-kube-api-access-24hhv\") pod \"iptables-alerter-w4qb2\" (UID: \"821107a9-9583-4e63-869d-bff90d64af15\") " pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:27.092146 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.092107    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal" event={"ID":"bcc7400c82ada97bc7d9b4033908bae2","Type":"ContainerStarted","Data":"96f36bf60cefa259e3a23354e9763af29520f7b4eef25100146a0675e0f55da3"}
Jul 06 20:35:27.093436 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.093388    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal" event={"ID":"487ce3ac867cb6a29534fc53f0fccb26","Type":"ContainerStarted","Data":"424330a25388d43c0d0a1ce2719046896dacabaaa2b41b02fe2f53dfad56da71"}
Jul 06 20:35:27.171108 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171072    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.171235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171119    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6njq\" (UniqueName: \"kubernetes.io/projected/1b599d48-d632-4337-9961-be63b99793e6-kube-api-access-w6njq\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.171235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171150    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12201c1c-c44f-4ce1-8d69-228ae73bd969-kubelet-config\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:27.171235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171175    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-tuned\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.171235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171200    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-tmp\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.171235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171223    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-var-lib-openvswitch\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171236    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171246    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-cni-netd\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171293    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-cni-netd\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171293    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40b3a754-3e30-4941-8978-ffe85a952f3a-env-overrides\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171334    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-registration-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171357    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-device-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171380    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-var-lib-cni-bin\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171404    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8x7\" (UniqueName: \"kubernetes.io/projected/4df480a7-e4bb-46d9-87d6-e07815d4f87b-kube-api-access-wr8x7\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171426    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-host\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171448    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-cnibin\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.171498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171473    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-run-k8s-cni-cncf-io\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171500    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-conf-dir\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171529    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-run-multus-certs\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171551    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-sysctl-d\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171574    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-run-systemd\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171589    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-var-lib-cni-bin\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171600    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-etc-openvswitch\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171635    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-var-lib-kubelet\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171648    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-etc-openvswitch\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171662    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171717    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-cnibin\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.171752    2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171760    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-run-k8s-cni-cncf-io\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171793    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-var-lib-openvswitch\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171810    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-conf-dir\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.171817 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.171822    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs podName:4df480a7-e4bb-46d9-87d6-e07815d4f87b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:27.67179279 +0000 UTC m=+3.013695502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs") pod "network-metrics-daemon-4zbdm" (UID: "4df480a7-e4bb-46d9-87d6-e07815d4f87b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171873    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-run-multus-certs\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171877    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12201c1c-c44f-4ce1-8d69-228ae73bd969-kubelet-config\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171889    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-registration-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171926    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-run-systemd\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171940    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-device-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171977    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-var-lib-kubelet\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.171751    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40b3a754-3e30-4941-8978-ffe85a952f3a-env-overrides\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172115    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-host\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172206    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-var-lib-kubelet\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172237    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/448af9b4-632e-4e75-991a-f1a5138b9a52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172255    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-systemd-units\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172269    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f3513df-fc03-42e5-86fc-bedae9f21fc0-serviceca\") pod \"node-ca-vz5nr\" (UID: \"4f3513df-fc03-42e5-86fc-bedae9f21fc0\") " pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.172332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172295    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172344    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-run-netns\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172373    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-var-lib-kubelet\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172378    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-system-cni-dir\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172415    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-run-netns\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172450    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-system-cni-dir\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172488    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-sys-fs\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172528    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-systemd-units\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172602    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-sysctl-d\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172747    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-cnibin\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172774    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-modprobe-d\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172783    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f3513df-fc03-42e5-86fc-bedae9f21fc0-serviceca\") pod \"node-ca-vz5nr\" (UID: \"4f3513df-fc03-42e5-86fc-bedae9f21fc0\") " pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172830    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-sysctl-conf\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172836    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-cnibin\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172870    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/448af9b4-632e-4e75-991a-f1a5138b9a52-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172871    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-modprobe-d\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172884    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-sys-fs\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.172976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172917    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/448af9b4-632e-4e75-991a-f1a5138b9a52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172930    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8cd\" (UniqueName: \"kubernetes.io/projected/448af9b4-632e-4e75-991a-f1a5138b9a52-kube-api-access-2h8cd\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172977    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-sysctl-conf\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.172980    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59-konnectivity-ca\") pod \"konnectivity-agent-466g2\" (UID: \"2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59\") " pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173021    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-run\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173104    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-cni-bin\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173132    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40b3a754-3e30-4941-8978-ffe85a952f3a-ovnkube-config\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173190    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-cni-bin\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173204    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-run\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173235    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40b3a754-3e30-4941-8978-ffe85a952f3a-ovnkube-script-lib\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173279    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f3513df-fc03-42e5-86fc-bedae9f21fc0-host\") pod \"node-ca-vz5nr\" (UID: \"4f3513df-fc03-42e5-86fc-bedae9f21fc0\") " pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173303    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-kubernetes\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173343    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-os-release\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173370    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-run-netns\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173380    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/448af9b4-632e-4e75-991a-f1a5138b9a52-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173393    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-cni-dir\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173431    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f3513df-fc03-42e5-86fc-bedae9f21fc0-host\") pod \"node-ca-vz5nr\" (UID: \"4f3513df-fc03-42e5-86fc-bedae9f21fc0\") " pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173457    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-os-release\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.173856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173473    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-kubernetes\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173496    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-run-netns\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173562    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59-konnectivity-ca\") pod \"konnectivity-agent-466g2\" (UID: \"2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59\") " pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173594    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-etc-kubernetes\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173639    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40b3a754-3e30-4941-8978-ffe85a952f3a-ovnkube-config\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173651    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-cni-dir\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173660    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-etc-kubernetes\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173662    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-systemd\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173713    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-socket-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173740    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59-agent-certs\") pod \"konnectivity-agent-466g2\" (UID: \"2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59\") " pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173765    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-os-release\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173781    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-systemd\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173791    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-slash\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173817    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-node-log\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173844    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b876p\" (UniqueName: \"kubernetes.io/projected/40b3a754-3e30-4941-8978-ffe85a952f3a-kube-api-access-b876p\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173868    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12201c1c-c44f-4ce1-8d69-228ae73bd969-dbus\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173884    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-os-release\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173893    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-run-openvswitch\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.174597 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173899    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-node-log\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173925    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-system-cni-dir\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173940    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-run-openvswitch\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173959    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-var-lib-cni-multus\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173978    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-slash\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173985    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174012    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-run-ovn\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174034    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174044    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-system-cni-dir\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174057    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-socket-dir-parent\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.173984    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40b3a754-3e30-4941-8978-ffe85a952f3a-ovnkube-script-lib\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174102    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174105    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-host-var-lib-cni-multus\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174112    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12201c1c-c44f-4ce1-8d69-228ae73bd969-dbus\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174128    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174154    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40b3a754-3e30-4941-8978-ffe85a952f3a-ovn-node-metrics-cert\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174158    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.175693 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174185    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-daemon-config\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174211    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-sysconfig\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.174217    2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174232    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-sys\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.174274    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret podName:12201c1c-c44f-4ce1-8d69-228ae73bd969 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:27.674255875 +0000 UTC m=+3.016158628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret") pod "global-pull-secret-syncer-p68f6" (UID: "12201c1c-c44f-4ce1-8d69-228ae73bd969") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174276    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/448af9b4-632e-4e75-991a-f1a5138b9a52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174286    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-sys\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174307    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppsp\" (UniqueName: \"kubernetes.io/projected/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-kube-api-access-dppsp\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174338    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/448af9b4-632e-4e75-991a-f1a5138b9a52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174364    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-kubelet\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174388    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7mw2\" (UniqueName: \"kubernetes.io/projected/4f3513df-fc03-42e5-86fc-bedae9f21fc0-kube-api-access-z7mw2\") pod \"node-ca-vz5nr\" (UID: \"4f3513df-fc03-42e5-86fc-bedae9f21fc0\") " pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174415    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174441    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktk8\" (UniqueName: \"kubernetes.io/projected/9f19dafe-c8e6-4053-9d18-542a67692a26-kube-api-access-wktk8\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174465    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f19dafe-c8e6-4053-9d18-542a67692a26-cni-binary-copy\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174488    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-hostroot\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174508    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-lib-modules\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174531    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-log-socket\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.176588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174571    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174602    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-log-socket\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174627    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-sysconfig\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174659    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-socket-dir-parent\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174695    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174719    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9f19dafe-c8e6-4053-9d18-542a67692a26-multus-daemon-config\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174773    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9f19dafe-c8e6-4053-9d18-542a67692a26-hostroot\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174881    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-lib-modules\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.174937    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-host-kubelet\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.175134    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f19dafe-c8e6-4053-9d18-542a67692a26-cni-binary-copy\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.175198    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40b3a754-3e30-4941-8978-ffe85a952f3a-run-ovn\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.175372    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/448af9b4-632e-4e75-991a-f1a5138b9a52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.175431    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-etc-tuned\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.177462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.175467    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-tmp\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.178817 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.178795    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:27.178922 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.178824    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:27.178922 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.178839    2541 projected.go:196] Error preparing data for projected volume kube-api-access-2tqpj for pod openshift-network-diagnostics/network-check-target-2vksd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:27.178922 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.178891    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj podName:a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb nodeName:}" failed. No retries permitted until 2026-07-06 20:35:27.678874023 +0000 UTC m=+3.020776727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2tqpj" (UniqueName: "kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj") pod "network-check-target-2vksd" (UID: "a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:27.179716 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.179686    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40b3a754-3e30-4941-8978-ffe85a952f3a-ovn-node-metrics-cert\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.180386 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.180366    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6njq\" (UniqueName: \"kubernetes.io/projected/1b599d48-d632-4337-9961-be63b99793e6-kube-api-access-w6njq\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.180879 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.180855    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59-agent-certs\") pod \"konnectivity-agent-466g2\" (UID: \"2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59\") " pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:27.181353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.181329    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b599d48-d632-4337-9961-be63b99793e6-socket-dir\") pod \"aws-ebs-csi-driver-node-x87cm\" (UID: \"1b599d48-d632-4337-9961-be63b99793e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.182071 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.182021    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8x7\" (UniqueName: \"kubernetes.io/projected/4df480a7-e4bb-46d9-87d6-e07815d4f87b-kube-api-access-wr8x7\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:27.183118 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.183095    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7mw2\" (UniqueName: \"kubernetes.io/projected/4f3513df-fc03-42e5-86fc-bedae9f21fc0-kube-api-access-z7mw2\") pod \"node-ca-vz5nr\" (UID: \"4f3513df-fc03-42e5-86fc-bedae9f21fc0\") " pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.183587 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.183567    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b876p\" (UniqueName: \"kubernetes.io/projected/40b3a754-3e30-4941-8978-ffe85a952f3a-kube-api-access-b876p\") pod \"ovnkube-node-gpn7x\" (UID: \"40b3a754-3e30-4941-8978-ffe85a952f3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.183830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.183792    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8cd\" (UniqueName: \"kubernetes.io/projected/448af9b4-632e-4e75-991a-f1a5138b9a52-kube-api-access-2h8cd\") pod \"multus-additional-cni-plugins-t7wgp\" (UID: \"448af9b4-632e-4e75-991a-f1a5138b9a52\") " pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.183830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.183813    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktk8\" (UniqueName: \"kubernetes.io/projected/9f19dafe-c8e6-4053-9d18-542a67692a26-kube-api-access-wktk8\") pod \"multus-fh5b5\" (UID: \"9f19dafe-c8e6-4053-9d18-542a67692a26\") " pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.184314 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.184296    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppsp\" (UniqueName: \"kubernetes.io/projected/6ffdc1c3-89a0-4ac3-8c58-6125740d752e-kube-api-access-dppsp\") pod \"tuned-r89gp\" (UID: \"6ffdc1c3-89a0-4ac3-8c58-6125740d752e\") " pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.270077 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.269986    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w4qb2"
Jul 06 20:35:27.279072 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.279039    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-784f2"
Jul 06 20:35:27.289806 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.289783    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vz5nr"
Jul 06 20:35:27.295637 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.295615    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm"
Jul 06 20:35:27.303477 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.303458    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:27.310319 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.310296    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fh5b5"
Jul 06 20:35:27.316297 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.316278    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r89gp"
Jul 06 20:35:27.322210 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.322193    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t7wgp"
Jul 06 20:35:27.328027 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.328013    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:27.678008 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.677939    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:27.678173 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.678023    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:27.678173 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.678103    2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:27.678173 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.678167    2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:27.678173 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.678174    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret podName:12201c1c-c44f-4ce1-8d69-228ae73bd969 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:28.678159752 +0000 UTC m=+4.020062449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret") pod "global-pull-secret-syncer-p68f6" (UID: "12201c1c-c44f-4ce1-8d69-228ae73bd969") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:27.678337 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.678209    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs podName:4df480a7-e4bb-46d9-87d6-e07815d4f87b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:28.678198176 +0000 UTC m=+4.020100874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs") pod "network-metrics-daemon-4zbdm" (UID: "4df480a7-e4bb-46d9-87d6-e07815d4f87b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:27.778605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:27.778574    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:27.778800 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.778776    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:27.778862 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.778836    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:27.778862 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.778852    2541 projected.go:196] Error preparing data for projected volume kube-api-access-2tqpj for pod openshift-network-diagnostics/network-check-target-2vksd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:27.778936 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:27.778922    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj podName:a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb nodeName:}" failed. No retries permitted until 2026-07-06 20:35:28.77890069 +0000 UTC m=+4.120803390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tqpj" (UniqueName: "kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj") pod "network-check-target-2vksd" (UID: "a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:28.002451 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.002421    2541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:27 +0000 UTC" deadline="2028-01-18 05:09:25.659295671 +0000 UTC"
Jul 06 20:35:28.002451 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.002447    2541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13448h33m57.656851757s"
Jul 06 20:35:28.073808 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:28.073786    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821107a9_9583_4e63_869d_bff90d64af15.slice/crio-90af83499bd6024299810bc627f4720f5eb24ac849ac97037b97b48b69e78dce WatchSource:0}: Error finding container 90af83499bd6024299810bc627f4720f5eb24ac849ac97037b97b48b69e78dce: Status 404 returned error can't find the container with id 90af83499bd6024299810bc627f4720f5eb24ac849ac97037b97b48b69e78dce
Jul 06 20:35:28.074159 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:28.074141    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448af9b4_632e_4e75_991a_f1a5138b9a52.slice/crio-e6f5db3e3cb02d5d9360e37f547e3969fbd856a281a12c20f95da399a0e222fb WatchSource:0}: Error finding container e6f5db3e3cb02d5d9360e37f547e3969fbd856a281a12c20f95da399a0e222fb: Status 404 returned error can't find the container with id e6f5db3e3cb02d5d9360e37f547e3969fbd856a281a12c20f95da399a0e222fb
Jul 06 20:35:28.075239 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:28.075216    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee4fbe5_4e1b_46c7_8f1b_526edcfdad59.slice/crio-284e03dfd0ba0ff27c6d89c2743d7a95a820437e2eedfd240417faf11d30fc45 WatchSource:0}: Error finding container 284e03dfd0ba0ff27c6d89c2743d7a95a820437e2eedfd240417faf11d30fc45: Status 404 returned error can't find the container with id 284e03dfd0ba0ff27c6d89c2743d7a95a820437e2eedfd240417faf11d30fc45
Jul 06 20:35:28.076861 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:28.076837    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffdc1c3_89a0_4ac3_8c58_6125740d752e.slice/crio-a098c425e336e5dc47fe9bde452574c883c8db0b83f6b816eba01dd4a51e3811 WatchSource:0}: Error finding container a098c425e336e5dc47fe9bde452574c883c8db0b83f6b816eba01dd4a51e3811: Status 404 returned error can't find the container with id a098c425e336e5dc47fe9bde452574c883c8db0b83f6b816eba01dd4a51e3811
Jul 06 20:35:28.078017 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:28.077997    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b3a754_3e30_4941_8978_ffe85a952f3a.slice/crio-bf10ec0ffe4b6bf34db8c0dbedafda83ecbc13b78b6f4637903972092fa0ed03 WatchSource:0}: Error finding container bf10ec0ffe4b6bf34db8c0dbedafda83ecbc13b78b6f4637903972092fa0ed03: Status 404 returned error can't find the container with id bf10ec0ffe4b6bf34db8c0dbedafda83ecbc13b78b6f4637903972092fa0ed03
Jul 06 20:35:28.079445 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:35:28.079404    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b599d48_d632_4337_9961_be63b99793e6.slice/crio-b9176a4c4153f46ac42af7e5b61e3ee1871653b7a0b30e8c51390bcc738f947f WatchSource:0}: Error finding container b9176a4c4153f46ac42af7e5b61e3ee1871653b7a0b30e8c51390bcc738f947f: Status 404 returned error can't find the container with id b9176a4c4153f46ac42af7e5b61e3ee1871653b7a0b30e8c51390bcc738f947f
Jul 06 20:35:28.088054 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.088036    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:28.088054 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.088050    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:28.088231 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.088149    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:28.088267 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.088219    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:28.096788 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.096765    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vz5nr" event={"ID":"4f3513df-fc03-42e5-86fc-bedae9f21fc0","Type":"ContainerStarted","Data":"8d14912405466806efbd9eed69e8c77c454bbc53d029558b059188f324e4d822"}
Jul 06 20:35:28.099101 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.099074    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-784f2" event={"ID":"73332827-2d68-4a90-ba2c-d3ed85704330","Type":"ContainerStarted","Data":"9eae6bd9dad95fc4da894a743c0bf4c4df6a379dec8e584986778584b82ffee1"}
Jul 06 20:35:28.100523 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.100485    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-466g2" event={"ID":"2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59","Type":"ContainerStarted","Data":"284e03dfd0ba0ff27c6d89c2743d7a95a820437e2eedfd240417faf11d30fc45"}
Jul 06 20:35:28.102363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.102314    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w4qb2" event={"ID":"821107a9-9583-4e63-869d-bff90d64af15","Type":"ContainerStarted","Data":"90af83499bd6024299810bc627f4720f5eb24ac849ac97037b97b48b69e78dce"}
Jul 06 20:35:28.104116 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.104041    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm" event={"ID":"1b599d48-d632-4337-9961-be63b99793e6","Type":"ContainerStarted","Data":"b9176a4c4153f46ac42af7e5b61e3ee1871653b7a0b30e8c51390bcc738f947f"}
Jul 06 20:35:28.105910 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.105874    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" event={"ID":"40b3a754-3e30-4941-8978-ffe85a952f3a","Type":"ContainerStarted","Data":"bf10ec0ffe4b6bf34db8c0dbedafda83ecbc13b78b6f4637903972092fa0ed03"}
Jul 06 20:35:28.110628 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.110439    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r89gp" event={"ID":"6ffdc1c3-89a0-4ac3-8c58-6125740d752e","Type":"ContainerStarted","Data":"a098c425e336e5dc47fe9bde452574c883c8db0b83f6b816eba01dd4a51e3811"}
Jul 06 20:35:28.111656 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.111637    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7wgp" event={"ID":"448af9b4-632e-4e75-991a-f1a5138b9a52","Type":"ContainerStarted","Data":"e6f5db3e3cb02d5d9360e37f547e3969fbd856a281a12c20f95da399a0e222fb"}
Jul 06 20:35:28.112846 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.112825    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fh5b5" event={"ID":"9f19dafe-c8e6-4053-9d18-542a67692a26","Type":"ContainerStarted","Data":"cf2aa2ef56375a6c3aa9ee699a70654974334327c34ec0ddb47dcf5a1f92522f"}
Jul 06 20:35:28.686782 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.686676    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:28.686782 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.686772    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:28.687034 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.686912    2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:28.687034 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.686977    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret podName:12201c1c-c44f-4ce1-8d69-228ae73bd969 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:30.686957914 +0000 UTC m=+6.028860617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret") pod "global-pull-secret-syncer-p68f6" (UID: "12201c1c-c44f-4ce1-8d69-228ae73bd969") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:28.687452 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.687413    2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:28.687551 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.687472    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs podName:4df480a7-e4bb-46d9-87d6-e07815d4f87b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:30.6874561 +0000 UTC m=+6.029358813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs") pod "network-metrics-daemon-4zbdm" (UID: "4df480a7-e4bb-46d9-87d6-e07815d4f87b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:28.787531 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:28.787495    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:28.787700 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.787684    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:28.787761 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.787712    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:28.787761 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.787726    2541 projected.go:196] Error preparing data for projected volume kube-api-access-2tqpj for pod openshift-network-diagnostics/network-check-target-2vksd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:28.787879 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:28.787782    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj podName:a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb nodeName:}" failed. No retries permitted until 2026-07-06 20:35:30.787762972 +0000 UTC m=+6.129665687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tqpj" (UniqueName: "kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj") pod "network-check-target-2vksd" (UID: "a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:29.002950 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:29.002882    2541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:30:27 +0000 UTC" deadline="2027-12-02 01:29:52.504219172 +0000 UTC"
Jul 06 20:35:29.002950 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:29.002948    2541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12316h54m23.501275114s"
Jul 06 20:35:29.090063 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:29.090030    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:29.090269 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:29.090190    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:29.143063 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:29.143019    2541 generic.go:356] "Generic (PLEG): container finished" podID="bcc7400c82ada97bc7d9b4033908bae2" containerID="de4d5ee100d7cfffcb555f50121126d3aa8191f6ed662591cfb3a0a03b0e744c" exitCode=0
Jul 06 20:35:29.143247 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:29.143146    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal" event={"ID":"bcc7400c82ada97bc7d9b4033908bae2","Type":"ContainerDied","Data":"de4d5ee100d7cfffcb555f50121126d3aa8191f6ed662591cfb3a0a03b0e744c"}
Jul 06 20:35:29.154865 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:29.154835    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal" event={"ID":"487ce3ac867cb6a29534fc53f0fccb26","Type":"ContainerStarted","Data":"ada6d9b09f6e5aacde4052d751d62b7fcb892076fb5e70fff0da227ac573b41a"}
Jul 06 20:35:30.088226 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:30.087862    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:30.088226 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.088011    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:30.088873 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:30.088532    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:30.088873 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.088660    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:30.178973 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:30.178761    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal" event={"ID":"bcc7400c82ada97bc7d9b4033908bae2","Type":"ContainerStarted","Data":"5b6029d3308fc2bb328255eb4c25348dbba1b93894b402929f0f502d5eef709d"}
Jul 06 20:35:30.203474 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:30.203412    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-39.ec2.internal" podStartSLOduration=3.203389833 podStartE2EDuration="3.203389833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:29.177904001 +0000 UTC m=+4.519806719" watchObservedRunningTime="2026-07-06 20:35:30.203389833 +0000 UTC m=+5.545292554"
Jul 06 20:35:30.702266 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:30.702224    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:30.702457 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:30.702327    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:30.702546 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.702505    2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:30.702604 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.702579    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret podName:12201c1c-c44f-4ce1-8d69-228ae73bd969 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:34.702556259 +0000 UTC m=+10.044458967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret") pod "global-pull-secret-syncer-p68f6" (UID: "12201c1c-c44f-4ce1-8d69-228ae73bd969") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:30.703215 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.703190    2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:30.703329 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.703255    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs podName:4df480a7-e4bb-46d9-87d6-e07815d4f87b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:34.703236503 +0000 UTC m=+10.045139223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs") pod "network-metrics-daemon-4zbdm" (UID: "4df480a7-e4bb-46d9-87d6-e07815d4f87b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:30.802798 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:30.802759    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:30.803047 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.803019    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:30.803047 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.803042    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:30.803195 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.803058    2541 projected.go:196] Error preparing data for projected volume kube-api-access-2tqpj for pod openshift-network-diagnostics/network-check-target-2vksd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:30.803195 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:30.803153    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj podName:a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb nodeName:}" failed. No retries permitted until 2026-07-06 20:35:34.803124882 +0000 UTC m=+10.145027585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tqpj" (UniqueName: "kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj") pod "network-check-target-2vksd" (UID: "a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:31.088238 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:31.088175    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:31.088809 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:31.088330    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:32.087415 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:32.087381    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:32.087604 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:32.087545    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:32.089133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:32.087380    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:32.089469 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:32.089377    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:33.089603 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:33.089568    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:33.090061 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:33.089746    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:34.087610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:34.087570    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:34.087772 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:34.087578    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:34.087772 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.087718    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:34.087772 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.087763    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:34.734449 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:34.734409    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:34.734969 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:34.734474    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:34.734969 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.734589    2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:34.734969 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.734659    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs podName:4df480a7-e4bb-46d9-87d6-e07815d4f87b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:42.73464096 +0000 UTC m=+18.076543675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs") pod "network-metrics-daemon-4zbdm" (UID: "4df480a7-e4bb-46d9-87d6-e07815d4f87b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:34.735173 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.735001    2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:34.735173 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.735037    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret podName:12201c1c-c44f-4ce1-8d69-228ae73bd969 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:42.735027034 +0000 UTC m=+18.076929731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret") pod "global-pull-secret-syncer-p68f6" (UID: "12201c1c-c44f-4ce1-8d69-228ae73bd969") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:34.835312 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:34.835274    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:34.835502 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.835440    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:34.835502 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.835463    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:34.835502 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.835474    2541 projected.go:196] Error preparing data for projected volume kube-api-access-2tqpj for pod openshift-network-diagnostics/network-check-target-2vksd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:34.835672 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:34.835544    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj podName:a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb nodeName:}" failed. No retries permitted until 2026-07-06 20:35:42.835524731 +0000 UTC m=+18.177427484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tqpj" (UniqueName: "kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj") pod "network-check-target-2vksd" (UID: "a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:35.089950 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:35.089397    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:35.089950 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:35.089532    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:36.087330 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:36.087291    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:36.087821 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:36.087432    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:36.087885 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:36.087825    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:36.087941 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:36.087913    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:37.088377 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:37.088286    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:37.088835 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:37.088422    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:38.087940 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:38.087906    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:38.088164 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:38.087957    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:38.088164 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:38.088038    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:38.088274 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:38.088175    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:39.088036 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:39.087991    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:39.094466 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:39.088160    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:40.087918 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:40.087884    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:40.088126 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:40.087888    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:40.088126 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:40.088006    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:40.088504 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:40.088121    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:41.087196 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:41.087166    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:41.087376 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:41.087276    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:42.088087 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:42.088040    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:42.088543 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:42.088039    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:42.088543 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.088172    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:42.088543 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.088274    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:42.805017 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:42.804982    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:42.805202 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:42.805058    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:42.805202 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.805146    2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:42.805202 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.805146    2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:42.805364 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.805211    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret podName:12201c1c-c44f-4ce1-8d69-228ae73bd969 nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.805194148 +0000 UTC m=+34.147096843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret") pod "global-pull-secret-syncer-p68f6" (UID: "12201c1c-c44f-4ce1-8d69-228ae73bd969") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:42.805364 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.805226    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs podName:4df480a7-e4bb-46d9-87d6-e07815d4f87b nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.805220061 +0000 UTC m=+34.147122756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs") pod "network-metrics-daemon-4zbdm" (UID: "4df480a7-e4bb-46d9-87d6-e07815d4f87b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:42.905935 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:42.905897    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:42.906134 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.906110    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:42.906134 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.906128    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:42.906223 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.906138    2541 projected.go:196] Error preparing data for projected volume kube-api-access-2tqpj for pod openshift-network-diagnostics/network-check-target-2vksd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:42.906223 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:42.906196    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj podName:a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb nodeName:}" failed. No retries permitted until 2026-07-06 20:35:58.906182234 +0000 UTC m=+34.248084944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tqpj" (UniqueName: "kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj") pod "network-check-target-2vksd" (UID: "a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:43.087861 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:43.087778    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:43.088030 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:43.087952    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:44.087724 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:44.087650    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:44.088070 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:44.087668    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:44.088070 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:44.087788    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:44.088070 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:44.087826    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:45.088348 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:45.088306    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:45.088769 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:45.088421    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:46.087792 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:46.087753    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:46.087959 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:46.087769    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:46.087959 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:46.087892    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:46.088102 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:46.087985    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:47.087382 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:47.087328    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:47.087850 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:47.087496    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:48.087447 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.087114    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:48.088201 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.087130    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:48.088201 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:48.087522    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:48.088201 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:48.087611    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:48.220141 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.220111    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vz5nr" event={"ID":"4f3513df-fc03-42e5-86fc-bedae9f21fc0","Type":"ContainerStarted","Data":"d05d7e4d23471ac43ea9e27edb3d3c41d37b6bf987a8edea61ec35edfe4acc18"}
Jul 06 20:35:48.221328 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.221297    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-784f2" event={"ID":"73332827-2d68-4a90-ba2c-d3ed85704330","Type":"ContainerStarted","Data":"38b995bfd9653183d776be5ee44b23fd1e756711344687faab8fc42ad9988342"}
Jul 06 20:35:48.222605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.222584    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-466g2" event={"ID":"2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59","Type":"ContainerStarted","Data":"be5bb33a9d40991d7841e9b8aca1f16347da5b696d41c2c255df061c669a7120"}
Jul 06 20:35:48.223938 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.223914    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm" event={"ID":"1b599d48-d632-4337-9961-be63b99793e6","Type":"ContainerStarted","Data":"9ab394bccb1fd4552b112bc5fd99a86f29910e322f64df1b4f2da98f56467473"}
Jul 06 20:35:48.226073 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.225975    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" event={"ID":"40b3a754-3e30-4941-8978-ffe85a952f3a","Type":"ContainerStarted","Data":"5b8d323f1b421c892ccb0e38829d538b944db78d6fc0e9ce3fdf5ec27753c9f3"}
Jul 06 20:35:48.226136 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.226119    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" event={"ID":"40b3a754-3e30-4941-8978-ffe85a952f3a","Type":"ContainerStarted","Data":"12aa235d5b0ed2f9233f79302b11af114f563ecf6e84644177c1257627aadaf4"}
Jul 06 20:35:48.226193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.226180    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" event={"ID":"40b3a754-3e30-4941-8978-ffe85a952f3a","Type":"ContainerStarted","Data":"4fd9545c6fa3462cfd9de55fd405353d59913c1970ec34bfb938866f5eadd16a"}
Jul 06 20:35:48.226250 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.226238    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" event={"ID":"40b3a754-3e30-4941-8978-ffe85a952f3a","Type":"ContainerStarted","Data":"43ca47960a5b553061c346d2281ef3dce44f22b60bdd1f3f60a5cbdc07ccdc32"}
Jul 06 20:35:48.227654 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.227633    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r89gp" event={"ID":"6ffdc1c3-89a0-4ac3-8c58-6125740d752e","Type":"ContainerStarted","Data":"98c0b09321e50ba7f782552c80f257815b82d4c069c56da97442c91db7e52679"}
Jul 06 20:35:48.229124 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.229102    2541 generic.go:356] "Generic (PLEG): container finished" podID="448af9b4-632e-4e75-991a-f1a5138b9a52" containerID="7be64aafb7b9be3299c7aa0a7f6bc0ecaaeb1eb26004ce937e33c18042f5243e" exitCode=0
Jul 06 20:35:48.229225 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.229164    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7wgp" event={"ID":"448af9b4-632e-4e75-991a-f1a5138b9a52","Type":"ContainerDied","Data":"7be64aafb7b9be3299c7aa0a7f6bc0ecaaeb1eb26004ce937e33c18042f5243e"}
Jul 06 20:35:48.230537 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.230519    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fh5b5" event={"ID":"9f19dafe-c8e6-4053-9d18-542a67692a26","Type":"ContainerStarted","Data":"da32752536be0d28e119c81fedb4d9970cf0a5389e428eb6a939e99cce62723b"}
Jul 06 20:35:48.237198 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.237164    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vz5nr" podStartSLOduration=3.6289632750000003 podStartE2EDuration="23.237152163s" totalImagesPullingTime="19.608188888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:48.23672233 +0000 UTC m=+23.578625047" watchObservedRunningTime="2026-07-06 20:35:48.237152163 +0000 UTC m=+23.579054915"
Jul 06 20:35:48.237264 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.237230    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-39.ec2.internal" podStartSLOduration=21.23722473 podStartE2EDuration="21.23722473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:30.204747404 +0000 UTC m=+5.546650118" watchObservedRunningTime="2026-07-06 20:35:48.23722473 +0000 UTC m=+23.579127446"
Jul 06 20:35:48.266274 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.266233    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-784f2" podStartSLOduration=3.650638265 podStartE2EDuration="23.266218066s" totalImagesPullingTime="19.615579801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:48.251213465 +0000 UTC m=+23.593116181" watchObservedRunningTime="2026-07-06 20:35:48.266218066 +0000 UTC m=+23.608120801"
Jul 06 20:35:48.266485 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.266467    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-466g2" podStartSLOduration=2.660859443 podStartE2EDuration="22.266462406s" totalImagesPullingTime="19.605602963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:48.266189615 +0000 UTC m=+23.608092333" watchObservedRunningTime="2026-07-06 20:35:48.266462406 +0000 UTC m=+23.608365124"
Jul 06 20:35:48.283538 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.283501    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r89gp" podStartSLOduration=2.661562725 podStartE2EDuration="22.283487859s" totalImagesPullingTime="19.621925134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:48.283258574 +0000 UTC m=+23.625161290" watchObservedRunningTime="2026-07-06 20:35:48.283487859 +0000 UTC m=+23.625390575"
Jul 06 20:35:48.299513 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:48.299476    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fh5b5" podStartSLOduration=2.587527276 podStartE2EDuration="22.299462794s" totalImagesPullingTime="19.711935518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:48.298943483 +0000 UTC m=+23.640846201" watchObservedRunningTime="2026-07-06 20:35:48.299462794 +0000 UTC m=+23.641365510"
Jul 06 20:35:49.088187 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:49.088153    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:49.088664 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:49.088300    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:49.235292 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:49.235255    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w4qb2" event={"ID":"821107a9-9583-4e63-869d-bff90d64af15","Type":"ContainerStarted","Data":"c7702a792bc6eee0f68e4e13689d368d025e5035cbbbf1f6fcbae9e61a42bb34"}
Jul 06 20:35:49.238828 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:49.238755    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" event={"ID":"40b3a754-3e30-4941-8978-ffe85a952f3a","Type":"ContainerStarted","Data":"3adef1c17869493c973268def5c4d5d3ccde50e218e3835d04664d2cb86b298a"}
Jul 06 20:35:49.238828 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:49.238824    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" event={"ID":"40b3a754-3e30-4941-8978-ffe85a952f3a","Type":"ContainerStarted","Data":"331cd8ed7d11703f3549b25482d5e7c0bd5ae28780bb958e8c0c43a18c3f7e87"}
Jul 06 20:35:49.249960 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:49.249907    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w4qb2" podStartSLOduration=4.63281887 podStartE2EDuration="24.2498879s" totalImagesPullingTime="19.61706903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:49.249560711 +0000 UTC m=+24.591463427" watchObservedRunningTime="2026-07-06 20:35:49.2498879 +0000 UTC m=+24.591790620"
Jul 06 20:35:49.544091 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:49.544028    2541 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 20:35:49.672370 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:49.672285    2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:49.673012 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:49.672991    2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:50.043100 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:50.042526    2541 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:35:49.544085713Z","UUID":"3a9ee068-423e-4d90-9aca-85ec5cacf547","Handler":null,"Name":"","Endpoint":""}
Jul 06 20:35:50.044352 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:50.044333    2541 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 20:35:50.044486 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:50.044372    2541 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 20:35:50.087805 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:50.087776    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:50.087977 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:50.087818    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:50.087977 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:50.087902    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:50.088119 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:50.088063    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:50.243872 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:50.243832    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm" event={"ID":"1b599d48-d632-4337-9961-be63b99793e6","Type":"ContainerStarted","Data":"feebb2a9ba16d6852e4225613848a33a2ee59f1e20d75a9efd8fd19cdffc0b29"}
Jul 06 20:35:51.088430 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:51.088180    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:51.088614 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:51.088584    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:51.250133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:51.250093    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" event={"ID":"40b3a754-3e30-4941-8978-ffe85a952f3a","Type":"ContainerStarted","Data":"da13f5b271a9865936cb68e97c74f29c2aead79cbab9385bbcec9ff784a9de80"}
Jul 06 20:35:51.252403 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:51.252368    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm" event={"ID":"1b599d48-d632-4337-9961-be63b99793e6","Type":"ContainerStarted","Data":"38ead254d4c30f3fad815e038140a18eb6b4ef756f0d56ff68fa6fd28714ee4f"}
Jul 06 20:35:51.252403 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:51.252399    2541 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 06 20:35:51.270668 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:51.270606    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87cm" podStartSLOduration=4.228224247 podStartE2EDuration="26.270585655s" totalImagesPullingTime="22.042361408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:51.269516354 +0000 UTC m=+26.611419070" watchObservedRunningTime="2026-07-06 20:35:51.270585655 +0000 UTC m=+26.612488372"
Jul 06 20:35:51.915980 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:51.915945    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:51.916630 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:51.916606    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-466g2"
Jul 06 20:35:52.087595 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:52.087556    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:52.087595 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:52.087581    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:52.087791 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:52.087690    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:52.087834 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:52.087802    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:53.087896 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:53.087646    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:53.088732 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:53.087912    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:53.259759 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:53.259716    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" event={"ID":"40b3a754-3e30-4941-8978-ffe85a952f3a","Type":"ContainerStarted","Data":"5ceb0bbc92e429da2283bea68e25a105200cb6836eadca4418649090af4c8ffc"}
Jul 06 20:35:53.259981 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:53.259964    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:53.261504 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:53.261476    2541 generic.go:356] "Generic (PLEG): container finished" podID="448af9b4-632e-4e75-991a-f1a5138b9a52" containerID="949f0818c59dd1654e3bb29784501c01ad8cbbd212aa106784ca3338dac3077c" exitCode=0
Jul 06 20:35:53.261630 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:53.261558    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7wgp" event={"ID":"448af9b4-632e-4e75-991a-f1a5138b9a52","Type":"ContainerDied","Data":"949f0818c59dd1654e3bb29784501c01ad8cbbd212aa106784ca3338dac3077c"}
Jul 06 20:35:53.275892 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:53.275873    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:53.285524 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:53.285487    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x" podStartSLOduration=7.615377679 podStartE2EDuration="27.285475593s" totalImagesPullingTime="19.670097914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:35:53.285203897 +0000 UTC m=+28.627106613" watchObservedRunningTime="2026-07-06 20:35:53.285475593 +0000 UTC m=+28.627378309"
Jul 06 20:35:54.087817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.087783    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:54.088067 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.087826    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:54.088067 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:54.087916    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:54.088442 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:54.088098    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:54.264316 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.264286    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:54.264548 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.264368    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:54.279734 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.279706    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:35:54.566682 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.566657    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p68f6"]
Jul 06 20:35:54.566799 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.566789    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:54.566887 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:54.566869    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:54.570066 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.570008    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zbdm"]
Jul 06 20:35:54.570189 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.570139    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:54.570250 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:54.570228    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:54.570867 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.570789    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2vksd"]
Jul 06 20:35:54.570967 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:54.570884    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:54.571189 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:54.570965    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:55.268271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:55.268065    2541 generic.go:356] "Generic (PLEG): container finished" podID="448af9b4-632e-4e75-991a-f1a5138b9a52" containerID="9b2395ac0e70281da9d57620ba27500c86a156cfbb8c76544275a049556a8674" exitCode=0
Jul 06 20:35:55.268738 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:55.268148    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7wgp" event={"ID":"448af9b4-632e-4e75-991a-f1a5138b9a52","Type":"ContainerDied","Data":"9b2395ac0e70281da9d57620ba27500c86a156cfbb8c76544275a049556a8674"}
Jul 06 20:35:56.088094 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:56.088040    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:56.088257 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:56.088068    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:56.088257 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:56.088171    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:56.088257 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:56.088178    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:56.088257 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:56.088236    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:56.088664 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:56.088313    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:57.276312 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:57.276275    2541 generic.go:356] "Generic (PLEG): container finished" podID="448af9b4-632e-4e75-991a-f1a5138b9a52" containerID="5a1ead404a99fb06f99d918e73b1d9084eb958943e0763187f398c0d02ce7339" exitCode=0
Jul 06 20:35:57.277153 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:57.276335    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7wgp" event={"ID":"448af9b4-632e-4e75-991a-f1a5138b9a52","Type":"ContainerDied","Data":"5a1ead404a99fb06f99d918e73b1d9084eb958943e0763187f398c0d02ce7339"}
Jul 06 20:35:58.087236 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:58.087201    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:58.087416 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.087333    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:35:58.087724 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:58.087707    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:58.087810 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.087794    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:35:58.087865 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:58.087841    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:58.087904 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.087886    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:35:58.823340 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:58.823296    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:35:58.823768 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:58.823361    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:35:58.823768 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.823487    2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:58.823768 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.823546    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs podName:4df480a7-e4bb-46d9-87d6-e07815d4f87b nodeName:}" failed. No retries permitted until 2026-07-06 20:36:30.823530082 +0000 UTC m=+66.165432777 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs") pod "network-metrics-daemon-4zbdm" (UID: "4df480a7-e4bb-46d9-87d6-e07815d4f87b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 20:35:58.823901 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.823869    2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:58.823935 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.823921    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret podName:12201c1c-c44f-4ce1-8d69-228ae73bd969 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:30.823906782 +0000 UTC m=+66.165809494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret") pod "global-pull-secret-syncer-p68f6" (UID: "12201c1c-c44f-4ce1-8d69-228ae73bd969") : object "kube-system"/"original-pull-secret" not registered
Jul 06 20:35:58.924650 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:35:58.924609    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:35:58.924839 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.924810    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 20:35:58.924839 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.924839    2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 20:35:58.924937 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.924851    2541 projected.go:196] Error preparing data for projected volume kube-api-access-2tqpj for pod openshift-network-diagnostics/network-check-target-2vksd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:35:58.924937 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:35:58.924908    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj podName:a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb nodeName:}" failed. No retries permitted until 2026-07-06 20:36:30.924892298 +0000 UTC m=+66.266795017 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2tqpj" (UniqueName: "kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj") pod "network-check-target-2vksd" (UID: "a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 20:36:00.088164 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.088126    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:36:00.088164 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.088164    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:36:00.088667 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.088164    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:36:00.088667 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:00.088246    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p68f6" podUID="12201c1c-c44f-4ce1-8d69-228ae73bd969"
Jul 06 20:36:00.088667 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:00.088342    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zbdm" podUID="4df480a7-e4bb-46d9-87d6-e07815d4f87b"
Jul 06 20:36:00.088667 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:00.088413    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vksd" podUID="a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb"
Jul 06 20:36:00.468906 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.468834    2541 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-39.ec2.internal" event="NodeReady"
Jul 06 20:36:00.469070 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.468997    2541 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 20:36:00.514124 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.514085    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7fbsp"]
Jul 06 20:36:00.519506 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.519483    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-55922"]
Jul 06 20:36:00.519645 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.519625    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.521485 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.521462    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5jj9\""
Jul 06 20:36:00.521633 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.521614    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 20:36:00.521745 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.521462    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 20:36:00.524020 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.523602    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55922"
Jul 06 20:36:00.524020 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.523907    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55922"]
Jul 06 20:36:00.525290 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.525269    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 20:36:00.525389 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.525271    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c2j6x\""
Jul 06 20:36:00.525539 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.525523    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 20:36:00.525677 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.525660    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 20:36:00.526723 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.526701    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7fbsp"]
Jul 06 20:36:00.640157 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.640122    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3721ac03-bf4f-477d-864a-f34cc9b9895c-cert\") pod \"ingress-canary-55922\" (UID: \"3721ac03-bf4f-477d-864a-f34cc9b9895c\") " pod="openshift-ingress-canary/ingress-canary-55922"
Jul 06 20:36:00.640339 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.640186    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-tmp-dir\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.640339 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.640265    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9jd\" (UniqueName: \"kubernetes.io/projected/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-kube-api-access-vr9jd\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.640339 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.640330    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-config-volume\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.640499 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.640374    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.640499 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.640435    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz4lp\" (UniqueName: \"kubernetes.io/projected/3721ac03-bf4f-477d-864a-f34cc9b9895c-kube-api-access-bz4lp\") pod \"ingress-canary-55922\" (UID: \"3721ac03-bf4f-477d-864a-f34cc9b9895c\") " pod="openshift-ingress-canary/ingress-canary-55922"
Jul 06 20:36:00.741398 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.741360    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz4lp\" (UniqueName: \"kubernetes.io/projected/3721ac03-bf4f-477d-864a-f34cc9b9895c-kube-api-access-bz4lp\") pod \"ingress-canary-55922\" (UID: \"3721ac03-bf4f-477d-864a-f34cc9b9895c\") " pod="openshift-ingress-canary/ingress-canary-55922"
Jul 06 20:36:00.741594 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.741416    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3721ac03-bf4f-477d-864a-f34cc9b9895c-cert\") pod \"ingress-canary-55922\" (UID: \"3721ac03-bf4f-477d-864a-f34cc9b9895c\") " pod="openshift-ingress-canary/ingress-canary-55922"
Jul 06 20:36:00.741594 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.741521    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-tmp-dir\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.741594 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.741573    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9jd\" (UniqueName: \"kubernetes.io/projected/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-kube-api-access-vr9jd\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.741753 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.741598    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-config-volume\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.741753 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.741629    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.741753 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:00.741730    2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:36:00.741889 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:00.741786    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls podName:22e65afc-1f02-48a2-b7ff-9213eb6cbff7 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:01.241767337 +0000 UTC m=+36.583670053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls") pod "dns-default-7fbsp" (UID: "22e65afc-1f02-48a2-b7ff-9213eb6cbff7") : secret "dns-default-metrics-tls" not found
Jul 06 20:36:00.742131 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.742108    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-tmp-dir\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.742698 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.742676    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-config-volume\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.746443 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.746285    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3721ac03-bf4f-477d-864a-f34cc9b9895c-cert\") pod \"ingress-canary-55922\" (UID: \"3721ac03-bf4f-477d-864a-f34cc9b9895c\") " pod="openshift-ingress-canary/ingress-canary-55922"
Jul 06 20:36:00.749647 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.749623    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9jd\" (UniqueName: \"kubernetes.io/projected/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-kube-api-access-vr9jd\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:00.749953 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.749929    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz4lp\" (UniqueName: \"kubernetes.io/projected/3721ac03-bf4f-477d-864a-f34cc9b9895c-kube-api-access-bz4lp\") pod \"ingress-canary-55922\" (UID: \"3721ac03-bf4f-477d-864a-f34cc9b9895c\") " pod="openshift-ingress-canary/ingress-canary-55922"
Jul 06 20:36:00.841162 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:00.841117    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-55922"
Jul 06 20:36:01.246333 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:01.246297    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:01.246771 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:01.246443    2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:36:01.246771 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:01.246530    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls podName:22e65afc-1f02-48a2-b7ff-9213eb6cbff7 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:02.246507001 +0000 UTC m=+37.588409701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls") pod "dns-default-7fbsp" (UID: "22e65afc-1f02-48a2-b7ff-9213eb6cbff7") : secret "dns-default-metrics-tls" not found
Jul 06 20:36:01.655347 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:01.655322    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-55922"]
Jul 06 20:36:01.668027 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:36:01.668001    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3721ac03_bf4f_477d_864a_f34cc9b9895c.slice/crio-32865eeac3a1162cc9a6af630e7f4cacab7b23e7e0b8f8f824a64e42ca3ffb51 WatchSource:0}: Error finding container 32865eeac3a1162cc9a6af630e7f4cacab7b23e7e0b8f8f824a64e42ca3ffb51: Status 404 returned error can't find the container with id 32865eeac3a1162cc9a6af630e7f4cacab7b23e7e0b8f8f824a64e42ca3ffb51
Jul 06 20:36:02.087501 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.087422    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:36:02.087501 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.087446    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:36:02.087719 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.087537    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:36:02.090509 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.090486    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:36:02.090688 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.090676    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzq7f\""
Jul 06 20:36:02.090809 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.090799    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:36:02.090943 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.090932    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:36:02.091078 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.091066    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:36:02.091194 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.091183    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hq9zz\""
Jul 06 20:36:02.254918 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.254883    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:02.255382 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:02.255066    2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:36:02.255382 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:02.255147    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls podName:22e65afc-1f02-48a2-b7ff-9213eb6cbff7 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:04.255127231 +0000 UTC m=+39.597029930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls") pod "dns-default-7fbsp" (UID: "22e65afc-1f02-48a2-b7ff-9213eb6cbff7") : secret "dns-default-metrics-tls" not found
Jul 06 20:36:02.292265 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.292158    2541 generic.go:356] "Generic (PLEG): container finished" podID="448af9b4-632e-4e75-991a-f1a5138b9a52" containerID="56af217e218d6bf8e72d7af78851cdc208aa61a2ef38ff5fc3099ce9488c2b9c" exitCode=0
Jul 06 20:36:02.292265 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.292204    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7wgp" event={"ID":"448af9b4-632e-4e75-991a-f1a5138b9a52","Type":"ContainerDied","Data":"56af217e218d6bf8e72d7af78851cdc208aa61a2ef38ff5fc3099ce9488c2b9c"}
Jul 06 20:36:02.293554 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:02.293521    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55922" event={"ID":"3721ac03-bf4f-477d-864a-f34cc9b9895c","Type":"ContainerStarted","Data":"32865eeac3a1162cc9a6af630e7f4cacab7b23e7e0b8f8f824a64e42ca3ffb51"}
Jul 06 20:36:03.300885 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:03.300628    2541 generic.go:356] "Generic (PLEG): container finished" podID="448af9b4-632e-4e75-991a-f1a5138b9a52" containerID="23cd5de08d29d104047464e017e77fc8c2be5d0191b7427b2f16143d10390d41" exitCode=0
Jul 06 20:36:03.301321 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:03.300798    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7wgp" event={"ID":"448af9b4-632e-4e75-991a-f1a5138b9a52","Type":"ContainerDied","Data":"23cd5de08d29d104047464e017e77fc8c2be5d0191b7427b2f16143d10390d41"}
Jul 06 20:36:04.271142 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:04.271114    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:04.271280 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:04.271237    2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:36:04.271339 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:04.271305    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls podName:22e65afc-1f02-48a2-b7ff-9213eb6cbff7 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:08.271286538 +0000 UTC m=+43.613189238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls") pod "dns-default-7fbsp" (UID: "22e65afc-1f02-48a2-b7ff-9213eb6cbff7") : secret "dns-default-metrics-tls" not found
Jul 06 20:36:04.304483 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:04.304453    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-55922" event={"ID":"3721ac03-bf4f-477d-864a-f34cc9b9895c","Type":"ContainerStarted","Data":"80b10f341f52a0590fe60723d7f45bd3d923e68d0ada326fcab4d8f6dc47cd44"}
Jul 06 20:36:04.307581 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:04.307556    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7wgp" event={"ID":"448af9b4-632e-4e75-991a-f1a5138b9a52","Type":"ContainerStarted","Data":"90edaa262991c242dd854f86167e5f7eb6fe4794ea5623149960dbe0abcc4b9f"}
Jul 06 20:36:04.339844 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:04.339804    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-55922" podStartSLOduration=2.151117812 podStartE2EDuration="4.339791806s" totalImagesPullingTime="2.188673994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:04.3196636 +0000 UTC m=+39.661566318" watchObservedRunningTime="2026-07-06 20:36:04.339791806 +0000 UTC m=+39.681694522"
Jul 06 20:36:04.340234 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:04.340205    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t7wgp" podStartSLOduration=7.611964755 podStartE2EDuration="38.340196385s" totalImagesPullingTime="30.72823163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:04.338466327 +0000 UTC m=+39.680369044" watchObservedRunningTime="2026-07-06 20:36:04.340196385 +0000 UTC m=+39.682099102"
Jul 06 20:36:08.296943 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:08.296901    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:08.297297 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:08.297058    2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:36:08.297297 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:08.297114    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls podName:22e65afc-1f02-48a2-b7ff-9213eb6cbff7 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:16.297100594 +0000 UTC m=+51.639003290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls") pod "dns-default-7fbsp" (UID: "22e65afc-1f02-48a2-b7ff-9213eb6cbff7") : secret "dns-default-metrics-tls" not found
Jul 06 20:36:16.350682 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:16.350654    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:16.351144 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:16.350798    2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:36:16.351144 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:16.350862    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls podName:22e65afc-1f02-48a2-b7ff-9213eb6cbff7 nodeName:}" failed. No retries permitted until 2026-07-06 20:36:32.350846922 +0000 UTC m=+67.692749617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls") pod "dns-default-7fbsp" (UID: "22e65afc-1f02-48a2-b7ff-9213eb6cbff7") : secret "dns-default-metrics-tls" not found
Jul 06 20:36:26.286822 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:26.286789    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpn7x"
Jul 06 20:36:30.840446 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.840406    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:36:30.840938 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.840475    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:36:30.842748 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.842730    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 20:36:30.842823 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.842777    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 20:36:30.851201 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:30.851179    2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 20:36:30.851303 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:30.851235    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs podName:4df480a7-e4bb-46d9-87d6-e07815d4f87b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:34.851221134 +0000 UTC m=+130.193123829 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs") pod "network-metrics-daemon-4zbdm" (UID: "4df480a7-e4bb-46d9-87d6-e07815d4f87b") : secret "metrics-daemon-secret" not found
Jul 06 20:36:30.854872 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.854853    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12201c1c-c44f-4ce1-8d69-228ae73bd969-original-pull-secret\") pod \"global-pull-secret-syncer-p68f6\" (UID: \"12201c1c-c44f-4ce1-8d69-228ae73bd969\") " pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:36:30.899931 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.899906    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p68f6"
Jul 06 20:36:30.941692 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.941662    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:36:30.945502 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.945470    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 20:36:30.955178 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.955153    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 20:36:30.966254 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:30.966224    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqpj\" (UniqueName: \"kubernetes.io/projected/a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb-kube-api-access-2tqpj\") pod \"network-check-target-2vksd\" (UID: \"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb\") " pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:36:31.022783 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:31.022754    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p68f6"]
Jul 06 20:36:31.026176 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:36:31.026145    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12201c1c_c44f_4ce1_8d69_228ae73bd969.slice/crio-305e8f18972c7aa76f359af8c099985ea48fa7621a5d432c09a5fed3929b7911 WatchSource:0}: Error finding container 305e8f18972c7aa76f359af8c099985ea48fa7621a5d432c09a5fed3929b7911: Status 404 returned error can't find the container with id 305e8f18972c7aa76f359af8c099985ea48fa7621a5d432c09a5fed3929b7911
Jul 06 20:36:31.215220 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:31.215152    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzq7f\""
Jul 06 20:36:31.224135 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:31.224116    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:36:31.341113 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:31.341084    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2vksd"]
Jul 06 20:36:31.344651 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:36:31.344624    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f06fd8_d8d0_4169_9fa1_dbded13ed0cb.slice/crio-de4c7e9a338f5f1ef89995e867e74fe450ec7787099ee72b5ff232ccd01c74c9 WatchSource:0}: Error finding container de4c7e9a338f5f1ef89995e867e74fe450ec7787099ee72b5ff232ccd01c74c9: Status 404 returned error can't find the container with id de4c7e9a338f5f1ef89995e867e74fe450ec7787099ee72b5ff232ccd01c74c9
Jul 06 20:36:31.375580 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:31.375538    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p68f6" event={"ID":"12201c1c-c44f-4ce1-8d69-228ae73bd969","Type":"ContainerStarted","Data":"305e8f18972c7aa76f359af8c099985ea48fa7621a5d432c09a5fed3929b7911"}
Jul 06 20:36:31.376524 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:31.376497    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2vksd" event={"ID":"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb","Type":"ContainerStarted","Data":"de4c7e9a338f5f1ef89995e867e74fe450ec7787099ee72b5ff232ccd01c74c9"}
Jul 06 20:36:32.450886 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:32.450846    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:36:32.451390 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:32.450994    2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:36:32.451390 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:36:32.451096    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls podName:22e65afc-1f02-48a2-b7ff-9213eb6cbff7 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:04.45107628 +0000 UTC m=+99.792978981 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls") pod "dns-default-7fbsp" (UID: "22e65afc-1f02-48a2-b7ff-9213eb6cbff7") : secret "dns-default-metrics-tls" not found
Jul 06 20:36:36.392989 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:36.392953    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p68f6" event={"ID":"12201c1c-c44f-4ce1-8d69-228ae73bd969","Type":"ContainerStarted","Data":"72932271b53f0bb65f57a246ca531b08c0fba54464d8209159e86c6307204cab"}
Jul 06 20:36:36.394365 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:36.394343    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2vksd" event={"ID":"a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb","Type":"ContainerStarted","Data":"9a21464f18bfb4b79127b92b5ddfd143a501f7056d01525bd5a9a3f6ed31c7b4"}
Jul 06 20:36:36.394497 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:36.394482    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:36:36.410372 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:36.410326    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p68f6" podStartSLOduration=65.841859563 podStartE2EDuration="1m10.410312579s" totalImagesPullingTime="4.568453016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:36.40972472 +0000 UTC m=+71.751627436" watchObservedRunningTime="2026-07-06 20:36:36.410312579 +0000 UTC m=+71.752215294"
Jul 06 20:36:36.428402 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:36:36.428352    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2vksd" podStartSLOduration=66.18291735 podStartE2EDuration="1m10.428338268s" totalImagesPullingTime="4.245420918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:36:36.428159811 +0000 UTC m=+71.770062528" watchObservedRunningTime="2026-07-06 20:36:36.428338268 +0000 UTC m=+71.770240989"
Jul 06 20:37:04.451407 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:04.451337    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:37:04.451728 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:04.451443    2541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 20:37:04.451728 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:04.451499    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls podName:22e65afc-1f02-48a2-b7ff-9213eb6cbff7 nodeName:}" failed. No retries permitted until 2026-07-06 20:38:08.45148602 +0000 UTC m=+163.793388716 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls") pod "dns-default-7fbsp" (UID: "22e65afc-1f02-48a2-b7ff-9213eb6cbff7") : secret "dns-default-metrics-tls" not found
Jul 06 20:37:07.398622 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:07.398593    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2vksd"
Jul 06 20:37:14.914433 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:14.914404    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"]
Jul 06 20:37:14.916722 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:14.916708    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:14.918546 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:14.918525    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 06 20:37:14.918930 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:14.918917    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2bfw9\""
Jul 06 20:37:14.919785 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:14.919769    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 20:37:14.920407 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:14.920392    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 20:37:14.920858 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:14.920840    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 06 20:37:14.928659 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:14.928631    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"]
Jul 06 20:37:15.010338 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.010312    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:15.010417 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.010344    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxgt\" (UniqueName: \"kubernetes.io/projected/71beb5de-c8d6-4dec-b5ac-8fc153f78270-kube-api-access-gbxgt\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:15.010417 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.010387    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/71beb5de-c8d6-4dec-b5ac-8fc153f78270-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:15.013306 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.013284    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959"]
Jul 06 20:37:15.015593 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.015580    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959"
Jul 06 20:37:15.017343 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.017323    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d5jr9\""
Jul 06 20:37:15.017343 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.017335    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:37:15.017472 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.017359    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 06 20:37:15.019944 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.019926    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"]
Jul 06 20:37:15.022450 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.022435    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:15.024402 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.024379    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959"]
Jul 06 20:37:15.024498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.024381    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 06 20:37:15.024498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.024491    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x6c2k\""
Jul 06 20:37:15.024681 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.024661    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 06 20:37:15.024769 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.024737    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:37:15.037534 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.037514    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"]
Jul 06 20:37:15.110752 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.110728    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:15.110846 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.110759    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxgt\" (UniqueName: \"kubernetes.io/projected/71beb5de-c8d6-4dec-b5ac-8fc153f78270-kube-api-access-gbxgt\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:15.110846 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.110786    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:15.110846 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.110826    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwzl\" (UniqueName: \"kubernetes.io/projected/546c165d-b822-40b8-8fae-7299137e126d-kube-api-access-8kwzl\") pod \"volume-data-source-validator-5d456fdbcd-f2959\" (UID: \"546c165d-b822-40b8-8fae-7299137e126d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959"
Jul 06 20:37:15.110947 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.110849    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlwpv\" (UniqueName: \"kubernetes.io/projected/f45164fc-fdfa-41e7-a574-9b75551b0408-kube-api-access-dlwpv\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:15.110947 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.110881    2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:15.110947 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.110892    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/71beb5de-c8d6-4dec-b5ac-8fc153f78270-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:15.110947 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.110942    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls podName:71beb5de-c8d6-4dec-b5ac-8fc153f78270 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:15.610924578 +0000 UTC m=+110.952827274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-qtl8n" (UID: "71beb5de-c8d6-4dec-b5ac-8fc153f78270") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:15.111498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.111481    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/71beb5de-c8d6-4dec-b5ac-8fc153f78270-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:15.115573 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.115544    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"]
Jul 06 20:37:15.118055 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.118035    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.119767 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.119740    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 06 20:37:15.119949 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.119933    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 06 20:37:15.120032 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.119970    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 06 20:37:15.120104 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.120076    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 06 20:37:15.120220 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.120203    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w8tbt\""
Jul 06 20:37:15.120535 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.120519    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c5579b66f-l4r7l"]
Jul 06 20:37:15.122847 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.122828    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.125255 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.125233    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bzrpw\""
Jul 06 20:37:15.125255 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.125247    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 20:37:15.125396 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.125251    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 20:37:15.125396 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.125241    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 20:37:15.128972 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.128950    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxgt\" (UniqueName: \"kubernetes.io/projected/71beb5de-c8d6-4dec-b5ac-8fc153f78270-kube-api-access-gbxgt\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:15.129815 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.129798    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"]
Jul 06 20:37:15.130848 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.130829    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 20:37:15.147190 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.147170    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c5579b66f-l4r7l"]
Jul 06 20:37:15.211236 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211191    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwzl\" (UniqueName: \"kubernetes.io/projected/546c165d-b822-40b8-8fae-7299137e126d-kube-api-access-8kwzl\") pod \"volume-data-source-validator-5d456fdbcd-f2959\" (UID: \"546c165d-b822-40b8-8fae-7299137e126d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959"
Jul 06 20:37:15.211236 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211215    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlwpv\" (UniqueName: \"kubernetes.io/projected/f45164fc-fdfa-41e7-a574-9b75551b0408-kube-api-access-dlwpv\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:15.211236 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211233    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c3f951c-98cf-4331-b0da-7370e678e6d2-ca-trust-extracted\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.211390 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211252    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-installation-pull-secrets\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.211390 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211268    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f07831f0-b8dd-4ff7-9808-ab07505f5f3a-config\") pod \"service-ca-operator-5d46d9ffd4-8wf5x\" (UID: \"f07831f0-b8dd-4ff7-9808-ab07505f5f3a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.211390 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211302    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-trusted-ca\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.211505 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211390    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-bound-sa-token\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.211505 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211406    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kk4b\" (UniqueName: \"kubernetes.io/projected/f07831f0-b8dd-4ff7-9808-ab07505f5f3a-kube-api-access-5kk4b\") pod \"service-ca-operator-5d46d9ffd4-8wf5x\" (UID: \"f07831f0-b8dd-4ff7-9808-ab07505f5f3a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.211505 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211427    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d9nt\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-kube-api-access-9d9nt\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.211505 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211455    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-image-registry-private-configuration\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.211505 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211475    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.211505 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211500    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-certificates\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.211728 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211555    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f07831f0-b8dd-4ff7-9808-ab07505f5f3a-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8wf5x\" (UID: \"f07831f0-b8dd-4ff7-9808-ab07505f5f3a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.211728 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.211577    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:15.211728 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.211661    2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:37:15.211728 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.211714    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls podName:f45164fc-fdfa-41e7-a574-9b75551b0408 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:15.711698508 +0000 UTC m=+111.053601204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bnkn6" (UID: "f45164fc-fdfa-41e7-a574-9b75551b0408") : secret "samples-operator-tls" not found
Jul 06 20:37:15.221615 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.221588    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlwpv\" (UniqueName: \"kubernetes.io/projected/f45164fc-fdfa-41e7-a574-9b75551b0408-kube-api-access-dlwpv\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:15.221750 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.221731    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwzl\" (UniqueName: \"kubernetes.io/projected/546c165d-b822-40b8-8fae-7299137e126d-kube-api-access-8kwzl\") pod \"volume-data-source-validator-5d456fdbcd-f2959\" (UID: \"546c165d-b822-40b8-8fae-7299137e126d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959"
Jul 06 20:37:15.312207 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312186    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c3f951c-98cf-4331-b0da-7370e678e6d2-ca-trust-extracted\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.312279 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312209    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-installation-pull-secrets\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.312279 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312224    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f07831f0-b8dd-4ff7-9808-ab07505f5f3a-config\") pod \"service-ca-operator-5d46d9ffd4-8wf5x\" (UID: \"f07831f0-b8dd-4ff7-9808-ab07505f5f3a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.312279 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312240    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-trusted-ca\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.312279 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312253    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-bound-sa-token\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.312279 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312268    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kk4b\" (UniqueName: \"kubernetes.io/projected/f07831f0-b8dd-4ff7-9808-ab07505f5f3a-kube-api-access-5kk4b\") pod \"service-ca-operator-5d46d9ffd4-8wf5x\" (UID: \"f07831f0-b8dd-4ff7-9808-ab07505f5f3a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.312494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312285    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d9nt\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-kube-api-access-9d9nt\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.312494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312311    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-image-registry-private-configuration\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.312494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312334    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.312494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312381    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-certificates\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.312494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312433    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f07831f0-b8dd-4ff7-9808-ab07505f5f3a-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8wf5x\" (UID: \"f07831f0-b8dd-4ff7-9808-ab07505f5f3a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.312729 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312689    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c3f951c-98cf-4331-b0da-7370e678e6d2-ca-trust-extracted\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.312785 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.312772    2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:37:15.312841 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.312786    2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c5579b66f-l4r7l: secret "image-registry-tls" not found
Jul 06 20:37:15.312841 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312831    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f07831f0-b8dd-4ff7-9808-ab07505f5f3a-config\") pod \"service-ca-operator-5d46d9ffd4-8wf5x\" (UID: \"f07831f0-b8dd-4ff7-9808-ab07505f5f3a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.312938 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.312921    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-certificates\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.313037 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.312989    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls podName:1c3f951c-98cf-4331-b0da-7370e678e6d2 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:15.812862529 +0000 UTC m=+111.154765227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls") pod "image-registry-6c5579b66f-l4r7l" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2") : secret "image-registry-tls" not found
Jul 06 20:37:15.313273 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.313243    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-trusted-ca\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.314566 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.314542    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-installation-pull-secrets\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.314665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.314640    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f07831f0-b8dd-4ff7-9808-ab07505f5f3a-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-8wf5x\" (UID: \"f07831f0-b8dd-4ff7-9808-ab07505f5f3a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.314742 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.314719    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-image-registry-private-configuration\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.320907 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.320885    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kk4b\" (UniqueName: \"kubernetes.io/projected/f07831f0-b8dd-4ff7-9808-ab07505f5f3a-kube-api-access-5kk4b\") pod \"service-ca-operator-5d46d9ffd4-8wf5x\" (UID: \"f07831f0-b8dd-4ff7-9808-ab07505f5f3a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.320980 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.320954    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-bound-sa-token\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.321089 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.321073    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d9nt\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-kube-api-access-9d9nt\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.323753 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.323740    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959"
Jul 06 20:37:15.426606 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.426581    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"
Jul 06 20:37:15.439618 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.439594    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959"]
Jul 06 20:37:15.444169 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:15.444132    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546c165d_b822_40b8_8fae_7299137e126d.slice/crio-289df67a20a17d4b6d9d77296d448eb679bd8020497339b63037cc5e895b6317 WatchSource:0}: Error finding container 289df67a20a17d4b6d9d77296d448eb679bd8020497339b63037cc5e895b6317: Status 404 returned error can't find the container with id 289df67a20a17d4b6d9d77296d448eb679bd8020497339b63037cc5e895b6317
Jul 06 20:37:15.479912 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.479833    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959" event={"ID":"546c165d-b822-40b8-8fae-7299137e126d","Type":"ContainerStarted","Data":"289df67a20a17d4b6d9d77296d448eb679bd8020497339b63037cc5e895b6317"}
Jul 06 20:37:15.543100 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.543076    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x"]
Jul 06 20:37:15.545629 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:15.545603    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07831f0_b8dd_4ff7_9808_ab07505f5f3a.slice/crio-e8f0238926353b2b4cb9feb3ba8f41ba4fd6912277fd7583821ce6ce2e50069d WatchSource:0}: Error finding container e8f0238926353b2b4cb9feb3ba8f41ba4fd6912277fd7583821ce6ce2e50069d: Status 404 returned error can't find the container with id e8f0238926353b2b4cb9feb3ba8f41ba4fd6912277fd7583821ce6ce2e50069d
Jul 06 20:37:15.615151 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.615127    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:15.615272 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.615258    2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:15.615323 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.615314    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls podName:71beb5de-c8d6-4dec-b5ac-8fc153f78270 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:16.615296919 +0000 UTC m=+111.957199614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-qtl8n" (UID: "71beb5de-c8d6-4dec-b5ac-8fc153f78270") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:15.716271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.716242    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:15.716377 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.716360    2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:37:15.716439 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.716432    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls podName:f45164fc-fdfa-41e7-a574-9b75551b0408 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:16.71641483 +0000 UTC m=+112.058317527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bnkn6" (UID: "f45164fc-fdfa-41e7-a574-9b75551b0408") : secret "samples-operator-tls" not found
Jul 06 20:37:15.816960 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:15.816931    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:15.817082 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.817033    2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:37:15.817082 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.817070    2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c5579b66f-l4r7l: secret "image-registry-tls" not found
Jul 06 20:37:15.817190 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:15.817121    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls podName:1c3f951c-98cf-4331-b0da-7370e678e6d2 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:16.817105113 +0000 UTC m=+112.159007823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls") pod "image-registry-6c5579b66f-l4r7l" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2") : secret "image-registry-tls" not found
Jul 06 20:37:16.483908 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:16.483861    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x" event={"ID":"f07831f0-b8dd-4ff7-9808-ab07505f5f3a","Type":"ContainerStarted","Data":"e8f0238926353b2b4cb9feb3ba8f41ba4fd6912277fd7583821ce6ce2e50069d"}
Jul 06 20:37:16.623924 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:16.623885    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:16.624132 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:16.624041    2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:16.624132 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:16.624105    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls podName:71beb5de-c8d6-4dec-b5ac-8fc153f78270 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:18.624091528 +0000 UTC m=+113.965994224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-qtl8n" (UID: "71beb5de-c8d6-4dec-b5ac-8fc153f78270") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:16.724877 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:16.724842    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:16.725021 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:16.724970    2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:37:16.725105 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:16.725032    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls podName:f45164fc-fdfa-41e7-a574-9b75551b0408 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:18.725013647 +0000 UTC m=+114.066916345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bnkn6" (UID: "f45164fc-fdfa-41e7-a574-9b75551b0408") : secret "samples-operator-tls" not found
Jul 06 20:37:16.826250 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:16.826220    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:16.826364 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:16.826324    2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:37:16.826364 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:16.826334    2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c5579b66f-l4r7l: secret "image-registry-tls" not found
Jul 06 20:37:16.826443 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:16.826377    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls podName:1c3f951c-98cf-4331-b0da-7370e678e6d2 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:18.826364213 +0000 UTC m=+114.168266909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls") pod "image-registry-6c5579b66f-l4r7l" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2") : secret "image-registry-tls" not found
Jul 06 20:37:17.488034 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:17.487997    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959" event={"ID":"546c165d-b822-40b8-8fae-7299137e126d","Type":"ContainerStarted","Data":"7280f0d43368e03b3616b3a6e67c0bb313f5cfe22d0b41a7e89d5905f3b86884"}
Jul 06 20:37:17.505608 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:17.505563    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-f2959" podStartSLOduration=1.2234407489999999 podStartE2EDuration="2.505549463s" totalImagesPullingTime="1.282108714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:17.504597598 +0000 UTC m=+112.846500315" watchObservedRunningTime="2026-07-06 20:37:17.505549463 +0000 UTC m=+112.847452179"
Jul 06 20:37:18.494186 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:18.494150    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x" event={"ID":"f07831f0-b8dd-4ff7-9808-ab07505f5f3a","Type":"ContainerStarted","Data":"94448f1702cf76a8588add442a28662ee66769bf515c6b3164d8f667bd893cd1"}
Jul 06 20:37:18.509060 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:18.508994    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x" podStartSLOduration=1.227175627 podStartE2EDuration="3.508975694s" totalImagesPullingTime="2.281800067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:18.508015459 +0000 UTC m=+113.849918176" watchObservedRunningTime="2026-07-06 20:37:18.508975694 +0000 UTC m=+113.850878411"
Jul 06 20:37:18.638119 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:18.638083    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:18.638292 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:18.638242    2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:18.638331 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:18.638306    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls podName:71beb5de-c8d6-4dec-b5ac-8fc153f78270 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:22.638288888 +0000 UTC m=+117.980191586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-qtl8n" (UID: "71beb5de-c8d6-4dec-b5ac-8fc153f78270") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:18.738876 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:18.738841    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:18.739025 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:18.738967    2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:37:18.739075 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:18.739026    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls podName:f45164fc-fdfa-41e7-a574-9b75551b0408 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:22.739007527 +0000 UTC m=+118.080910223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bnkn6" (UID: "f45164fc-fdfa-41e7-a574-9b75551b0408") : secret "samples-operator-tls" not found
Jul 06 20:37:18.839228 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:18.839161    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:18.839324 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:18.839286    2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:37:18.839324 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:18.839298    2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c5579b66f-l4r7l: secret "image-registry-tls" not found
Jul 06 20:37:18.839389 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:18.839348    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls podName:1c3f951c-98cf-4331-b0da-7370e678e6d2 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:22.839332245 +0000 UTC m=+118.181234942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls") pod "image-registry-6c5579b66f-l4r7l" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2") : secret "image-registry-tls" not found
Jul 06 20:37:19.320920 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.320888    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mskws"]
Jul 06 20:37:19.324171 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.324156    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.326118 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.326093    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 20:37:19.326227 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.326139    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qlpp5\""
Jul 06 20:37:19.326351 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.326332    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 20:37:19.326812 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.326796    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 20:37:19.326888 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.326856    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 20:37:19.334835 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.334814    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mskws"]
Jul 06 20:37:19.443275 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.443244    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.443428 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.443289    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22kgf\" (UniqueName: \"kubernetes.io/projected/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-api-access-22kgf\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.443428 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.443316    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-data-volume\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.443428 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.443344    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.443428 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.443397    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-crio-socket\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.544347 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.544322    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22kgf\" (UniqueName: \"kubernetes.io/projected/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-api-access-22kgf\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.544739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.544355    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-data-volume\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.544739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.544378    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.544739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.544477    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-crio-socket\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.544739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.544528    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.544739 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:19.544549    2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:37:19.544739 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:19.544620    2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:19.544739 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:19.544634    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:20.044611851 +0000 UTC m=+115.386514563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : configmap "kube-rbac-proxy" not found
Jul 06 20:37:19.544739 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:19.544670    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:20.044655184 +0000 UTC m=+115.386557895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:19.544739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.544684    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-crio-socket\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.545011 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.544874    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-data-volume\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:19.552374 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:19.552354    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22kgf\" (UniqueName: \"kubernetes.io/projected/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-api-access-22kgf\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:20.048245 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.048211    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:20.048411 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:20.048339    2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:37:20.048411 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.048348    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:20.048472 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:20.048418    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:21.048398112 +0000 UTC m=+116.390300811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : configmap "kube-rbac-proxy" not found
Jul 06 20:37:20.048515 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:20.048498    2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:20.048569 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:20.048559    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:21.048546517 +0000 UTC m=+116.390449212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:20.086331 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.086298    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb"]
Jul 06 20:37:20.089390 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.089375    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb"
Jul 06 20:37:20.091402 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.091381    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-brcpj\""
Jul 06 20:37:20.091581 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.091490    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 06 20:37:20.092023 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.092006    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 06 20:37:20.117368 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.116263    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb"]
Jul 06 20:37:20.148901 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.148872    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7lm\" (UniqueName: \"kubernetes.io/projected/2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3-kube-api-access-qn7lm\") pod \"migrator-d57d58cc9-dbbjb\" (UID: \"2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb"
Jul 06 20:37:20.250312 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.250270    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7lm\" (UniqueName: \"kubernetes.io/projected/2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3-kube-api-access-qn7lm\") pod \"migrator-d57d58cc9-dbbjb\" (UID: \"2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb"
Jul 06 20:37:20.257989 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.257952    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7lm\" (UniqueName: \"kubernetes.io/projected/2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3-kube-api-access-qn7lm\") pod \"migrator-d57d58cc9-dbbjb\" (UID: \"2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb"
Jul 06 20:37:20.397970 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.397893    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb"
Jul 06 20:37:20.520678 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:20.520649    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb"]
Jul 06 20:37:20.523734 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:20.523696    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e427c32_d1e4_4be7_a443_0ba6f0a5a5c3.slice/crio-3e56aa4e65613ca3f329964a1d0e981d333478cc9c0d5d78e8b974bc8a3a07d9 WatchSource:0}: Error finding container 3e56aa4e65613ca3f329964a1d0e981d333478cc9c0d5d78e8b974bc8a3a07d9: Status 404 returned error can't find the container with id 3e56aa4e65613ca3f329964a1d0e981d333478cc9c0d5d78e8b974bc8a3a07d9
Jul 06 20:37:21.056827 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:21.056789    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:21.057243 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:21.056909    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:21.057243 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:21.056924    2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:37:21.057243 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:21.056993    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:23.056972713 +0000 UTC m=+118.398875410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : configmap "kube-rbac-proxy" not found
Jul 06 20:37:21.057243 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:21.057073    2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:21.057243 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:21.057123    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:23.057108419 +0000 UTC m=+118.399011114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:21.102730 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:21.102702    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-784f2_73332827-2d68-4a90-ba2c-d3ed85704330/dns-node-resolver/0.log"
Jul 06 20:37:21.503367 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:21.503336    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb" event={"ID":"2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3","Type":"ContainerStarted","Data":"3e56aa4e65613ca3f329964a1d0e981d333478cc9c0d5d78e8b974bc8a3a07d9"}
Jul 06 20:37:22.501518 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:22.501495    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vz5nr_4f3513df-fc03-42e5-86fc-bedae9f21fc0/node-ca/0.log"
Jul 06 20:37:22.668707 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:22.668668    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:22.668884 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:22.668826    2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:22.668932 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:22.668889    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls podName:71beb5de-c8d6-4dec-b5ac-8fc153f78270 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:30.66887304 +0000 UTC m=+126.010775736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-qtl8n" (UID: "71beb5de-c8d6-4dec-b5ac-8fc153f78270") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:22.769816 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:22.769725    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:22.769967 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:22.769897    2541 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 06 20:37:22.770032 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:22.769975    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls podName:f45164fc-fdfa-41e7-a574-9b75551b0408 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:30.769954484 +0000 UTC m=+126.111857195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bnkn6" (UID: "f45164fc-fdfa-41e7-a574-9b75551b0408") : secret "samples-operator-tls" not found
Jul 06 20:37:22.870992 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:22.870956    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:22.871175 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:22.871093    2541 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 20:37:22.871175 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:22.871113    2541 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c5579b66f-l4r7l: secret "image-registry-tls" not found
Jul 06 20:37:22.871175 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:22.871167    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls podName:1c3f951c-98cf-4331-b0da-7370e678e6d2 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:30.871152714 +0000 UTC m=+126.213055409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls") pod "image-registry-6c5579b66f-l4r7l" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2") : secret "image-registry-tls" not found
Jul 06 20:37:22.902876 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:22.902848    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-55922_3721ac03-bf4f-477d-864a-f34cc9b9895c/serve-healthcheck-canary/0.log"
Jul 06 20:37:23.072362 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:23.072280    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:23.072495 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:23.072382    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:23.072495 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:23.072414    2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:37:23.072495 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:23.072475    2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:23.072495 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:23.072485    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:27.072467776 +0000 UTC m=+122.414370474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : configmap "kube-rbac-proxy" not found
Jul 06 20:37:23.072673 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:23.072512    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:27.072501817 +0000 UTC m=+122.414404542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:23.509796 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:23.509761    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb" event={"ID":"2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3","Type":"ContainerStarted","Data":"5bd6134e4cbfd072a7bb3208f4a27ff9f050d1ad9b25e18edc58f93c347f347b"}
Jul 06 20:37:23.510168 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:23.509819    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb" event={"ID":"2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3","Type":"ContainerStarted","Data":"2ed5c577e031bbbd17d27cf0033753562fab92d9fca9404f81bf002e7e95e064"}
Jul 06 20:37:23.523723 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:23.523674    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-dbbjb" podStartSLOduration=1.568764244 podStartE2EDuration="3.523660213s" totalImagesPullingTime="1.954895969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:23.523236325 +0000 UTC m=+118.865139043" watchObservedRunningTime="2026-07-06 20:37:23.523660213 +0000 UTC m=+118.865562930"
Jul 06 20:37:27.103362 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:27.103331    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:27.103842 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:27.103419    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:27.103842 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:27.103458    2541 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 20:37:27.103842 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:27.103513    2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:27.103842 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:27.103535    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:35.103513712 +0000 UTC m=+130.445416407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : configmap "kube-rbac-proxy" not found
Jul 06 20:37:27.103842 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:27.103554    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:35.10354385 +0000 UTC m=+130.445446546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:30.730154 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:30.730117    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:30.730539 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:30.730229    2541 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:30.730539 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:30.730275    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls podName:71beb5de-c8d6-4dec-b5ac-8fc153f78270 nodeName:}" failed. No retries permitted until 2026-07-06 20:37:46.730262039 +0000 UTC m=+142.072164734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-qtl8n" (UID: "71beb5de-c8d6-4dec-b5ac-8fc153f78270") : secret "cluster-monitoring-operator-tls" not found
Jul 06 20:37:30.831087 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:30.831058    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:30.833362 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:30.833343    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f45164fc-fdfa-41e7-a574-9b75551b0408-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bnkn6\" (UID: \"f45164fc-fdfa-41e7-a574-9b75551b0408\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:30.931834 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:30.931813    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:30.933205 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:30.933190    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x6c2k\""
Jul 06 20:37:30.933925 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:30.933908    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") pod \"image-registry-6c5579b66f-l4r7l\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") " pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:30.941094 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:30.941076    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"
Jul 06 20:37:31.040859 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:31.040836    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bzrpw\""
Jul 06 20:37:31.049287 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:31.049262    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:31.059594 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:31.059574    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6"]
Jul 06 20:37:31.172269 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:31.172245    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c5579b66f-l4r7l"]
Jul 06 20:37:31.175404 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:31.175385    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3f951c_98cf_4331_b0da_7370e678e6d2.slice/crio-3ead77fd7fc1ba42c8f66a17a47f258d0c37201bfde69a8c5f36a95c9ff8eaed WatchSource:0}: Error finding container 3ead77fd7fc1ba42c8f66a17a47f258d0c37201bfde69a8c5f36a95c9ff8eaed: Status 404 returned error can't find the container with id 3ead77fd7fc1ba42c8f66a17a47f258d0c37201bfde69a8c5f36a95c9ff8eaed
Jul 06 20:37:31.533485 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:31.533450    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l" event={"ID":"1c3f951c-98cf-4331-b0da-7370e678e6d2","Type":"ContainerStarted","Data":"620054d25732787a2e8cb56c5dc2c7b467f2417e59b113316f5f3a00caaf2b3b"}
Jul 06 20:37:31.533649 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:31.533510    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l" event={"ID":"1c3f951c-98cf-4331-b0da-7370e678e6d2","Type":"ContainerStarted","Data":"3ead77fd7fc1ba42c8f66a17a47f258d0c37201bfde69a8c5f36a95c9ff8eaed"}
Jul 06 20:37:31.533649 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:31.533581    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:37:31.534662 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:31.534628    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6" event={"ID":"f45164fc-fdfa-41e7-a574-9b75551b0408","Type":"ContainerStarted","Data":"dce5cd8a7c6737b9c6e9adcd56344a737659268d024609c1ac5bfa7ae9085e76"}
Jul 06 20:37:31.551773 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:31.551718    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l" podStartSLOduration=16.551700765 podStartE2EDuration="16.551700765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:31.550038669 +0000 UTC m=+126.891941387" watchObservedRunningTime="2026-07-06 20:37:31.551700765 +0000 UTC m=+126.893603485"
Jul 06 20:37:33.542772 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:33.542739    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6" event={"ID":"f45164fc-fdfa-41e7-a574-9b75551b0408","Type":"ContainerStarted","Data":"d665a4198ac6a81a41ff1d9f693a338e2ddb4ad7ba3aa15935246c21426bda60"}
Jul 06 20:37:33.543120 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:33.542788    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6" event={"ID":"f45164fc-fdfa-41e7-a574-9b75551b0408","Type":"ContainerStarted","Data":"6d8fe461fc9ea080a0245680da943756d280f005f9d6af37e54e93aa2f239988"}
Jul 06 20:37:33.559688 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:33.559644    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bnkn6" podStartSLOduration=17.754109334 podStartE2EDuration="19.559630048s" totalImagesPullingTime="1.805520714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:33.559008489 +0000 UTC m=+128.900911206" watchObservedRunningTime="2026-07-06 20:37:33.559630048 +0000 UTC m=+128.901532765"
Jul 06 20:37:34.858439 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:34.858411    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:37:34.860518 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:34.860499    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4df480a7-e4bb-46d9-87d6-e07815d4f87b-metrics-certs\") pod \"network-metrics-daemon-4zbdm\" (UID: \"4df480a7-e4bb-46d9-87d6-e07815d4f87b\") " pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:37:35.109800 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:35.109738    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hq9zz\""
Jul 06 20:37:35.118368 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:35.118350    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zbdm"
Jul 06 20:37:35.160374 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:35.160342    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:35.160508 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:35.160393    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:35.160571 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:35.160515    2541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:35.160620 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:37:35.160590    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls podName:60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b nodeName:}" failed. No retries permitted until 2026-07-06 20:37:51.160569985 +0000 UTC m=+146.502472699 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mskws" (UID: "60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b") : secret "insights-runtime-extractor-tls" not found
Jul 06 20:37:35.160959 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:35.160930    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:35.236625 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:35.236593    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zbdm"]
Jul 06 20:37:35.240238 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:35.240215    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df480a7_e4bb_46d9_87d6_e07815d4f87b.slice/crio-fd2c6500c2e1fae9b1e72b9d088e98b836d319985f17ee5759f5bcec14660510 WatchSource:0}: Error finding container fd2c6500c2e1fae9b1e72b9d088e98b836d319985f17ee5759f5bcec14660510: Status 404 returned error can't find the container with id fd2c6500c2e1fae9b1e72b9d088e98b836d319985f17ee5759f5bcec14660510
Jul 06 20:37:35.549620 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:35.549593    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zbdm" event={"ID":"4df480a7-e4bb-46d9-87d6-e07815d4f87b","Type":"ContainerStarted","Data":"fd2c6500c2e1fae9b1e72b9d088e98b836d319985f17ee5759f5bcec14660510"}
Jul 06 20:37:36.554843 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:36.554814    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zbdm" event={"ID":"4df480a7-e4bb-46d9-87d6-e07815d4f87b","Type":"ContainerStarted","Data":"f13ed60ccaa08b8093242856c161d62416ac05d5309a56ad10275fa405b723a2"}
Jul 06 20:37:36.555217 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:36.554862    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zbdm" event={"ID":"4df480a7-e4bb-46d9-87d6-e07815d4f87b","Type":"ContainerStarted","Data":"ce64ba92ff0835fc01e886d6704ffd604c4fb27dfc8caf8c71cabde195a0478f"}
Jul 06 20:37:36.569653 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:36.569609    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4zbdm" podStartSLOduration=129.610795673 podStartE2EDuration="2m10.569595034s" totalImagesPullingTime="958.799361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:36.569234143 +0000 UTC m=+131.911136861" watchObservedRunningTime="2026-07-06 20:37:36.569595034 +0000 UTC m=+131.911497750"
Jul 06 20:37:44.568251 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.568215    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c5579b66f-l4r7l"]
Jul 06 20:37:44.570748 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.570717    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"]
Jul 06 20:37:44.574982 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.574964    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg"]
Jul 06 20:37:44.575125 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.575096    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"
Jul 06 20:37:44.577110 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.577075    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 06 20:37:44.577110 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.577079    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pgz44\""
Jul 06 20:37:44.577265 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.577144    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 06 20:37:44.582255 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.582237    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg"
Jul 06 20:37:44.584113 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.584092    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"]
Jul 06 20:37:44.584218 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.584165    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z85m8\""
Jul 06 20:37:44.586564 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.586547    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg"]
Jul 06 20:37:44.724013 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.723983    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s69rv\" (UniqueName: \"kubernetes.io/projected/08af6ed3-abe2-4380-9dd0-ea07dceadabc-kube-api-access-s69rv\") pod \"network-check-source-989f8b8dc-zk2lg\" (UID: \"08af6ed3-abe2-4380-9dd0-ea07dceadabc\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg"
Jul 06 20:37:44.724184 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.724017    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/56f05cee-4e8c-49e7-a8f8-0c957946f881-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-c6w28\" (UID: \"56f05cee-4e8c-49e7-a8f8-0c957946f881\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"
Jul 06 20:37:44.724184 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.724107    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/56f05cee-4e8c-49e7-a8f8-0c957946f881-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-c6w28\" (UID: \"56f05cee-4e8c-49e7-a8f8-0c957946f881\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"
Jul 06 20:37:44.825123 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.825062    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s69rv\" (UniqueName: \"kubernetes.io/projected/08af6ed3-abe2-4380-9dd0-ea07dceadabc-kube-api-access-s69rv\") pod \"network-check-source-989f8b8dc-zk2lg\" (UID: \"08af6ed3-abe2-4380-9dd0-ea07dceadabc\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg"
Jul 06 20:37:44.825123 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.825089    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/56f05cee-4e8c-49e7-a8f8-0c957946f881-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-c6w28\" (UID: \"56f05cee-4e8c-49e7-a8f8-0c957946f881\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"
Jul 06 20:37:44.825286 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.825125    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/56f05cee-4e8c-49e7-a8f8-0c957946f881-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-c6w28\" (UID: \"56f05cee-4e8c-49e7-a8f8-0c957946f881\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"
Jul 06 20:37:44.825674 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.825652    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/56f05cee-4e8c-49e7-a8f8-0c957946f881-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-c6w28\" (UID: \"56f05cee-4e8c-49e7-a8f8-0c957946f881\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"
Jul 06 20:37:44.827522 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.827505    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/56f05cee-4e8c-49e7-a8f8-0c957946f881-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-c6w28\" (UID: \"56f05cee-4e8c-49e7-a8f8-0c957946f881\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"
Jul 06 20:37:44.833346 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.833327    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s69rv\" (UniqueName: \"kubernetes.io/projected/08af6ed3-abe2-4380-9dd0-ea07dceadabc-kube-api-access-s69rv\") pod \"network-check-source-989f8b8dc-zk2lg\" (UID: \"08af6ed3-abe2-4380-9dd0-ea07dceadabc\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg"
Jul 06 20:37:44.891078 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.891059    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"
Jul 06 20:37:44.894990 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:44.894972    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg"
Jul 06 20:37:45.018407 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:45.018369    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-c6w28"]
Jul 06 20:37:45.022036 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:45.022009    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f05cee_4e8c_49e7_a8f8_0c957946f881.slice/crio-135e590715f64bab614b4ed1892625a4e5531139ddfc6c664754a3270b6a9a20 WatchSource:0}: Error finding container 135e590715f64bab614b4ed1892625a4e5531139ddfc6c664754a3270b6a9a20: Status 404 returned error can't find the container with id 135e590715f64bab614b4ed1892625a4e5531139ddfc6c664754a3270b6a9a20
Jul 06 20:37:45.036902 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:45.036880    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg"]
Jul 06 20:37:45.039550 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:45.039530    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08af6ed3_abe2_4380_9dd0_ea07dceadabc.slice/crio-3e4244401af27ab464f816c9557338e96ffbbb283be11d1ef9be29d0e89d7c02 WatchSource:0}: Error finding container 3e4244401af27ab464f816c9557338e96ffbbb283be11d1ef9be29d0e89d7c02: Status 404 returned error can't find the container with id 3e4244401af27ab464f816c9557338e96ffbbb283be11d1ef9be29d0e89d7c02
Jul 06 20:37:45.588368 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:45.588337    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg" event={"ID":"08af6ed3-abe2-4380-9dd0-ea07dceadabc","Type":"ContainerStarted","Data":"82825c1637fb619ea9e792fc054b3a404b44ae4e9243b1c38152eae7c976fddf"}
Jul 06 20:37:45.588747 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:45.588381    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg" event={"ID":"08af6ed3-abe2-4380-9dd0-ea07dceadabc","Type":"ContainerStarted","Data":"3e4244401af27ab464f816c9557338e96ffbbb283be11d1ef9be29d0e89d7c02"}
Jul 06 20:37:45.589455 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:45.589432    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28" event={"ID":"56f05cee-4e8c-49e7-a8f8-0c957946f881","Type":"ContainerStarted","Data":"135e590715f64bab614b4ed1892625a4e5531139ddfc6c664754a3270b6a9a20"}
Jul 06 20:37:45.603198 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:45.603161    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-zk2lg" podStartSLOduration=1.603148475 podStartE2EDuration="1.603148475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:45.601872264 +0000 UTC m=+140.943774981" watchObservedRunningTime="2026-07-06 20:37:45.603148475 +0000 UTC m=+140.945051191"
Jul 06 20:37:46.594037 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:46.593944    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28" event={"ID":"56f05cee-4e8c-49e7-a8f8-0c957946f881","Type":"ContainerStarted","Data":"974c9c42faf98d38c92abc82cb7ef8f35788dc5f72179c5eba5d2066912ff4cb"}
Jul 06 20:37:46.608108 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:46.608061    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-c6w28" podStartSLOduration=1.29761122 podStartE2EDuration="2.608032142s" totalImagesPullingTime="1.310420922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:46.606864937 +0000 UTC m=+141.948767654" watchObservedRunningTime="2026-07-06 20:37:46.608032142 +0000 UTC m=+141.949934858"
Jul 06 20:37:46.739709 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:46.739680    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:46.742174 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:46.742145    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/71beb5de-c8d6-4dec-b5ac-8fc153f78270-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-qtl8n\" (UID: \"71beb5de-c8d6-4dec-b5ac-8fc153f78270\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:47.026743 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:47.026722    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2bfw9\""
Jul 06 20:37:47.035897 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:47.035880    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"
Jul 06 20:37:47.155800 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:47.155771    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n"]
Jul 06 20:37:47.158375 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:47.158346    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71beb5de_c8d6_4dec_b5ac_8fc153f78270.slice/crio-ea00adb80696ff46747b124cb99fb4a4b7f729bf001f157e36ea9b285d8aa92c WatchSource:0}: Error finding container ea00adb80696ff46747b124cb99fb4a4b7f729bf001f157e36ea9b285d8aa92c: Status 404 returned error can't find the container with id ea00adb80696ff46747b124cb99fb4a4b7f729bf001f157e36ea9b285d8aa92c
Jul 06 20:37:47.597922 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:47.597897    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n" event={"ID":"71beb5de-c8d6-4dec-b5ac-8fc153f78270","Type":"ContainerStarted","Data":"ea00adb80696ff46747b124cb99fb4a4b7f729bf001f157e36ea9b285d8aa92c"}
Jul 06 20:37:49.532644 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.532612    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"]
Jul 06 20:37:49.535548 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.535528    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"
Jul 06 20:37:49.537302 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.537269    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 06 20:37:49.537928 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.537909    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2ldwt\""
Jul 06 20:37:49.545476 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.545458    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"]
Jul 06 20:37:49.605274 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.605252    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n" event={"ID":"71beb5de-c8d6-4dec-b5ac-8fc153f78270","Type":"ContainerStarted","Data":"fa58fba0d7f40e0ada8c5b590ade79c701a6362a8a57b7fdbe40722d15022a1a"}
Jul 06 20:37:49.620088 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.620026    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-qtl8n" podStartSLOduration=33.722678523 podStartE2EDuration="35.620013286s" totalImagesPullingTime="1.897334763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:49.618839432 +0000 UTC m=+144.960742149" watchObservedRunningTime="2026-07-06 20:37:49.620013286 +0000 UTC m=+144.961916002"
Jul 06 20:37:49.658467 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.658442    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce7ce0b2-dbcc-455d-b760-d518524d02a6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-zfqg2\" (UID: \"ce7ce0b2-dbcc-455d-b760-d518524d02a6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"
Jul 06 20:37:49.759659 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.759640    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce7ce0b2-dbcc-455d-b760-d518524d02a6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-zfqg2\" (UID: \"ce7ce0b2-dbcc-455d-b760-d518524d02a6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"
Jul 06 20:37:49.762166 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.762149    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce7ce0b2-dbcc-455d-b760-d518524d02a6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-zfqg2\" (UID: \"ce7ce0b2-dbcc-455d-b760-d518524d02a6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"
Jul 06 20:37:49.844149 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.844104    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"
Jul 06 20:37:49.960659 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:49.960634    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"]
Jul 06 20:37:49.963678 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:49.963655    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7ce0b2_dbcc_455d_b760_d518524d02a6.slice/crio-daa30448e2023e5f9a048020b5f570a906036ff0f47efd81517cad217eb088f0 WatchSource:0}: Error finding container daa30448e2023e5f9a048020b5f570a906036ff0f47efd81517cad217eb088f0: Status 404 returned error can't find the container with id daa30448e2023e5f9a048020b5f570a906036ff0f47efd81517cad217eb088f0
Jul 06 20:37:50.608980 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:50.608955    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2" event={"ID":"ce7ce0b2-dbcc-455d-b760-d518524d02a6","Type":"ContainerStarted","Data":"daa30448e2023e5f9a048020b5f570a906036ff0f47efd81517cad217eb088f0"}
Jul 06 20:37:51.171330 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.171293    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:51.174076 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.174026    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mskws\" (UID: \"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b\") " pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:51.434676 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.434626    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qlpp5\""
Jul 06 20:37:51.443169 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.443152    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mskws"
Jul 06 20:37:51.562805 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.562770    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mskws"]
Jul 06 20:37:51.565925 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:51.565898    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b4bbf9_ebfd_4771_b60f_6b85a1b53c5b.slice/crio-d697cba5dabcfafee640bb127ec1d153d07167b56156ac8fdd2608cd63d914cb WatchSource:0}: Error finding container d697cba5dabcfafee640bb127ec1d153d07167b56156ac8fdd2608cd63d914cb: Status 404 returned error can't find the container with id d697cba5dabcfafee640bb127ec1d153d07167b56156ac8fdd2608cd63d914cb
Jul 06 20:37:51.614077 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.614015    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2" event={"ID":"ce7ce0b2-dbcc-455d-b760-d518524d02a6","Type":"ContainerStarted","Data":"414fed7154bef75d5087f0d11bd69ebb0d32aa150252274e59326e0bb472276e"}
Jul 06 20:37:51.614420 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.614096    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"
Jul 06 20:37:51.615132 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.615112    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mskws" event={"ID":"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b","Type":"ContainerStarted","Data":"d697cba5dabcfafee640bb127ec1d153d07167b56156ac8fdd2608cd63d914cb"}
Jul 06 20:37:51.618506 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.618487    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2"
Jul 06 20:37:51.628120 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:51.628084    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-zfqg2" podStartSLOduration=1.365528066 podStartE2EDuration="2.628067427s" totalImagesPullingTime="1.262539361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:51.627331633 +0000 UTC m=+146.969234352" watchObservedRunningTime="2026-07-06 20:37:51.628067427 +0000 UTC m=+146.969970141"
Jul 06 20:37:52.597309 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.597276    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"]
Jul 06 20:37:52.600508 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.600478    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.602601 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.602577    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 06 20:37:52.602737 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.602703    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 06 20:37:52.603144 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.603125    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dcfd6\""
Jul 06 20:37:52.603252 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.603184    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 20:37:52.609498 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.609474    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"]
Jul 06 20:37:52.683730 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.683654    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd2afd06-71af-4154-98cf-711daa3d6ca1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.683730 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.683691    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gt9p\" (UniqueName: \"kubernetes.io/projected/cd2afd06-71af-4154-98cf-711daa3d6ca1-kube-api-access-6gt9p\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.684150 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.683736    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd2afd06-71af-4154-98cf-711daa3d6ca1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.684150 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.683810    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd2afd06-71af-4154-98cf-711daa3d6ca1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.784282 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.784259    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd2afd06-71af-4154-98cf-711daa3d6ca1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.784468 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.784298    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd2afd06-71af-4154-98cf-711daa3d6ca1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.784468 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.784345    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd2afd06-71af-4154-98cf-711daa3d6ca1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.784602 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.784513    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gt9p\" (UniqueName: \"kubernetes.io/projected/cd2afd06-71af-4154-98cf-711daa3d6ca1-kube-api-access-6gt9p\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.785219 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.785195    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd2afd06-71af-4154-98cf-711daa3d6ca1-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.787389 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.787367    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd2afd06-71af-4154-98cf-711daa3d6ca1-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.787512 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.787491    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd2afd06-71af-4154-98cf-711daa3d6ca1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.792683 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.792665    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gt9p\" (UniqueName: \"kubernetes.io/projected/cd2afd06-71af-4154-98cf-711daa3d6ca1-kube-api-access-6gt9p\") pod \"prometheus-operator-567f7b8b7c-5vvz4\" (UID: \"cd2afd06-71af-4154-98cf-711daa3d6ca1\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:52.912256 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:52.912190    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"
Jul 06 20:37:53.044798 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:53.044766    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4"]
Jul 06 20:37:54.574128 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:54.574081    2541 patch_prober.go:28] interesting pod/image-registry-6c5579b66f-l4r7l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 06 20:37:54.574504 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:54.574165    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l" podUID="1c3f951c-98cf-4331-b0da-7370e678e6d2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:37:54.746413 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:37:54.746379    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2afd06_71af_4154_98cf_711daa3d6ca1.slice/crio-330554a936ab000e57e874e543622130d89aa8c204b3af5667314b7918c35372 WatchSource:0}: Error finding container 330554a936ab000e57e874e543622130d89aa8c204b3af5667314b7918c35372: Status 404 returned error can't find the container with id 330554a936ab000e57e874e543622130d89aa8c204b3af5667314b7918c35372
Jul 06 20:37:55.630547 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:55.630512    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mskws" event={"ID":"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b","Type":"ContainerStarted","Data":"d6c05bca9b88130f5a81483e9f581436eaafcd01760748261d60afe1c13a1fcf"}
Jul 06 20:37:55.631956 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:55.631928    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4" event={"ID":"cd2afd06-71af-4154-98cf-711daa3d6ca1","Type":"ContainerStarted","Data":"330554a936ab000e57e874e543622130d89aa8c204b3af5667314b7918c35372"}
Jul 06 20:37:56.638380 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:56.638352    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mskws" event={"ID":"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b","Type":"ContainerStarted","Data":"8697ae020eace32232cb1e5fcf80f407ec55b97c6541a01e9d6f689ca50caccd"}
Jul 06 20:37:56.640336 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:56.640302    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4" event={"ID":"cd2afd06-71af-4154-98cf-711daa3d6ca1","Type":"ContainerStarted","Data":"cceba55e001ff1ffb3002ebd13488c9a1f0c79838b34a8591f7bbb9da09e02f9"}
Jul 06 20:37:57.645403 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:57.645369    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4" event={"ID":"cd2afd06-71af-4154-98cf-711daa3d6ca1","Type":"ContainerStarted","Data":"a862355096e38f75a076eada65a4e1bcd27392218e863eee75f30c698e4a57a1"}
Jul 06 20:37:57.675625 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:57.675580    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-5vvz4" podStartSLOduration=3.906238349 podStartE2EDuration="5.675564838s" totalImagesPullingTime="1.769326489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:57.672858201 +0000 UTC m=+153.014760917" watchObservedRunningTime="2026-07-06 20:37:57.675564838 +0000 UTC m=+153.017467568"
Jul 06 20:37:58.650018 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:58.649982    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mskws" event={"ID":"60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b","Type":"ContainerStarted","Data":"fd350361080501bec5d5cc4b09acb2b100eb87e1c2c303c47fe6819461e5c8bd"}
Jul 06 20:37:58.665599 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:58.665513    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mskws" podStartSLOduration=32.7871174 podStartE2EDuration="39.665496919s" totalImagesPullingTime="6.878379519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:37:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:37:58.665214017 +0000 UTC m=+154.007116733" watchObservedRunningTime="2026-07-06 20:37:58.665496919 +0000 UTC m=+154.007399636"
Jul 06 20:37:59.930141 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.930105    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"]
Jul 06 20:37:59.933321 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.933306    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:37:59.935453 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.935429    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 06 20:37:59.935520 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.935478    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 06 20:37:59.935520 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.935489    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f8hlg\""
Jul 06 20:37:59.944946 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.944921    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"]
Jul 06 20:37:59.956003 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.955983    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7l4gn"]
Jul 06 20:37:59.959241 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.959222    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:37:59.961277 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.961172    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 20:37:59.961277 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.961248    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 20:37:59.961406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.961177    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 20:37:59.961406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:37:59.961387    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k5595\""
Jul 06 20:38:00.042813 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.042795    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.042910 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.042831    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.042910 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.042871    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.042910 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.042893    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64twh\" (UniqueName: \"kubernetes.io/projected/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-kube-api-access-64twh\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.143963 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.143943    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4dbbef56-95f7-4188-a285-d4474048b1b9-sys\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.144095 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.143973    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-textfile\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.144095 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.143998    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.144095 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144013    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-wtmp\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.144095 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144033    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.144095 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144086    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.144303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144187    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64twh\" (UniqueName: \"kubernetes.io/projected/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-kube-api-access-64twh\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.144303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144248    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4dbbef56-95f7-4188-a285-d4474048b1b9-root\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.144303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144277    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-tls\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.144446 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144303    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5x2\" (UniqueName: \"kubernetes.io/projected/4dbbef56-95f7-4188-a285-d4474048b1b9-kube-api-access-kh5x2\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.144446 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144330    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.144446 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144371    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.144549 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144429    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dbbef56-95f7-4188-a285-d4474048b1b9-metrics-client-ca\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.144860 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.144845    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.146355 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.146339    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.146559 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.146540    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.151059 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.151025    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64twh\" (UniqueName: \"kubernetes.io/projected/62736c9f-e9ac-47bb-883c-df7a4bc52ea0-kube-api-access-64twh\") pod \"openshift-state-metrics-54cf96dfcf-7stxg\" (UID: \"62736c9f-e9ac-47bb-883c-df7a4bc52ea0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.242087 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.242067    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"
Jul 06 20:38:00.244860 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.244843    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4dbbef56-95f7-4188-a285-d4474048b1b9-root\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.244953 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.244878    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-tls\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.244953 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.244903    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5x2\" (UniqueName: \"kubernetes.io/projected/4dbbef56-95f7-4188-a285-d4474048b1b9-kube-api-access-kh5x2\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.244953 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.244928    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.244953 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.244942    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4dbbef56-95f7-4188-a285-d4474048b1b9-root\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245229 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.244955    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dbbef56-95f7-4188-a285-d4474048b1b9-metrics-client-ca\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245229 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.245027    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4dbbef56-95f7-4188-a285-d4474048b1b9-sys\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245229 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.245090    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-textfile\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245229 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.245120    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4dbbef56-95f7-4188-a285-d4474048b1b9-sys\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245229 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.245129    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-wtmp\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245229 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.245164    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245826 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.245245    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-wtmp\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245826 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.245443    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-textfile\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245826 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.245530    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dbbef56-95f7-4188-a285-d4474048b1b9-metrics-client-ca\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.245932 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.245848    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.246997 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.246977    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-tls\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.247287 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.247270    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4dbbef56-95f7-4188-a285-d4474048b1b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.255675 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.255653    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5x2\" (UniqueName: \"kubernetes.io/projected/4dbbef56-95f7-4188-a285-d4474048b1b9-kube-api-access-kh5x2\") pod \"node-exporter-7l4gn\" (UID: \"4dbbef56-95f7-4188-a285-d4474048b1b9\") " pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.267976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.267958    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7l4gn"
Jul 06 20:38:00.278731 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:38:00.278707    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbbef56_95f7_4188_a285_d4474048b1b9.slice/crio-f3a9de6bf158d9638b09c00b1b99b26465bed37c29f68b9f31a8c976c8a22c60 WatchSource:0}: Error finding container f3a9de6bf158d9638b09c00b1b99b26465bed37c29f68b9f31a8c976c8a22c60: Status 404 returned error can't find the container with id f3a9de6bf158d9638b09c00b1b99b26465bed37c29f68b9f31a8c976c8a22c60
Jul 06 20:38:00.373529 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.373501    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg"]
Jul 06 20:38:00.376466 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:38:00.376434    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62736c9f_e9ac_47bb_883c_df7a4bc52ea0.slice/crio-5adbb9e06699c430943b5b0ffa71b3fa8289977530b4407ea500e0c6d258f987 WatchSource:0}: Error finding container 5adbb9e06699c430943b5b0ffa71b3fa8289977530b4407ea500e0c6d258f987: Status 404 returned error can't find the container with id 5adbb9e06699c430943b5b0ffa71b3fa8289977530b4407ea500e0c6d258f987
Jul 06 20:38:00.659640 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.659613    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg" event={"ID":"62736c9f-e9ac-47bb-883c-df7a4bc52ea0","Type":"ContainerStarted","Data":"366500659d32ca03278269d6b5678bdf7b4aca4d2bfdfe1c7326b4a90ce10ebc"}
Jul 06 20:38:00.659780 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.659675    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg" event={"ID":"62736c9f-e9ac-47bb-883c-df7a4bc52ea0","Type":"ContainerStarted","Data":"cd6578853911c85c9733ccfb5c55ff4c68939d657ea6d3b9de568bc6a567bce5"}
Jul 06 20:38:00.659780 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.659706    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg" event={"ID":"62736c9f-e9ac-47bb-883c-df7a4bc52ea0","Type":"ContainerStarted","Data":"5adbb9e06699c430943b5b0ffa71b3fa8289977530b4407ea500e0c6d258f987"}
Jul 06 20:38:00.660780 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:00.660754    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7l4gn" event={"ID":"4dbbef56-95f7-4188-a285-d4474048b1b9","Type":"ContainerStarted","Data":"f3a9de6bf158d9638b09c00b1b99b26465bed37c29f68b9f31a8c976c8a22c60"}
Jul 06 20:38:01.666284 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:01.666018    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7l4gn" event={"ID":"4dbbef56-95f7-4188-a285-d4474048b1b9","Type":"ContainerStarted","Data":"f62f074ba77174c79da3145cfaf1158a6e72a63369dc95305dca9583501d0ca6"}
Jul 06 20:38:02.671908 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:02.671874    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg" event={"ID":"62736c9f-e9ac-47bb-883c-df7a4bc52ea0","Type":"ContainerStarted","Data":"17ca2742f64833349759a8e51f0b9e19f565ebe0d1ae535812c845cec63bfc04"}
Jul 06 20:38:02.673324 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:02.673298    2541 generic.go:356] "Generic (PLEG): container finished" podID="4dbbef56-95f7-4188-a285-d4474048b1b9" containerID="f62f074ba77174c79da3145cfaf1158a6e72a63369dc95305dca9583501d0ca6" exitCode=0
Jul 06 20:38:02.673432 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:02.673389    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7l4gn" event={"ID":"4dbbef56-95f7-4188-a285-d4474048b1b9","Type":"ContainerDied","Data":"f62f074ba77174c79da3145cfaf1158a6e72a63369dc95305dca9583501d0ca6"}
Jul 06 20:38:02.694857 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:02.694805    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7stxg" podStartSLOduration=2.390082611 podStartE2EDuration="3.694787506s" totalImagesPullingTime="1.304704895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:02.693978982 +0000 UTC m=+158.035881700" watchObservedRunningTime="2026-07-06 20:38:02.694787506 +0000 UTC m=+158.036690225"
Jul 06 20:38:03.534539 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:38:03.534498    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7fbsp" podUID="22e65afc-1f02-48a2-b7ff-9213eb6cbff7"
Jul 06 20:38:03.681305 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:03.681270    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7l4gn" event={"ID":"4dbbef56-95f7-4188-a285-d4474048b1b9","Type":"ContainerStarted","Data":"07890b69fb2b8aa6d241afcfe6c44c66b6fc482ead080e60f2dd4185c7434ebe"}
Jul 06 20:38:03.681305 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:03.681294    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:38:03.681741 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:03.681319    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7l4gn" event={"ID":"4dbbef56-95f7-4188-a285-d4474048b1b9","Type":"ContainerStarted","Data":"67d72d3d7e9a773aac14de865b1244a8f169c61dbe7ea3ec29b4095be0ec463d"}
Jul 06 20:38:03.699812 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:03.699774    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7l4gn" podStartSLOduration=3.405975018 podStartE2EDuration="4.69975919s" totalImagesPullingTime="1.218769339s" totalInitContainerRuntime="75.014833ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:03.699553166 +0000 UTC m=+159.041455886" watchObservedRunningTime="2026-07-06 20:38:03.69975919 +0000 UTC m=+159.041661907"
Jul 06 20:38:04.572549 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.572527    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:38:04.715480 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.715454    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"]
Jul 06 20:38:04.718654 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.718639    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:04.720592 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.720568    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 06 20:38:04.720742 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.720724    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dl8jd\""
Jul 06 20:38:04.728341 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.728322    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"]
Jul 06 20:38:04.881754 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.881690    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ljj\" (UniqueName: \"kubernetes.io/projected/2c319108-a62c-40fe-a619-0ca069679212-kube-api-access-f6ljj\") pod \"monitoring-plugin-d5dcff7cf-279vp\" (UID: \"2c319108-a62c-40fe-a619-0ca069679212\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:04.881913 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.881886    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c319108-a62c-40fe-a619-0ca069679212-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-279vp\" (UID: \"2c319108-a62c-40fe-a619-0ca069679212\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:04.982581 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.982560    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ljj\" (UniqueName: \"kubernetes.io/projected/2c319108-a62c-40fe-a619-0ca069679212-kube-api-access-f6ljj\") pod \"monitoring-plugin-d5dcff7cf-279vp\" (UID: \"2c319108-a62c-40fe-a619-0ca069679212\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:04.982655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.982624    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c319108-a62c-40fe-a619-0ca069679212-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-279vp\" (UID: \"2c319108-a62c-40fe-a619-0ca069679212\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:04.982724 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:38:04.982713    2541 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 06 20:38:04.982769 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:38:04.982754    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c319108-a62c-40fe-a619-0ca069679212-monitoring-plugin-cert podName:2c319108-a62c-40fe-a619-0ca069679212 nodeName:}" failed. No retries permitted until 2026-07-06 20:38:05.482741883 +0000 UTC m=+160.824644578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2c319108-a62c-40fe-a619-0ca069679212-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-279vp" (UID: "2c319108-a62c-40fe-a619-0ca069679212") : secret "monitoring-plugin-cert" not found
Jul 06 20:38:04.990335 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:04.990310    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ljj\" (UniqueName: \"kubernetes.io/projected/2c319108-a62c-40fe-a619-0ca069679212-kube-api-access-f6ljj\") pod \"monitoring-plugin-d5dcff7cf-279vp\" (UID: \"2c319108-a62c-40fe-a619-0ca069679212\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:05.486230 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:05.486204    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c319108-a62c-40fe-a619-0ca069679212-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-279vp\" (UID: \"2c319108-a62c-40fe-a619-0ca069679212\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:05.488652 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:05.488627    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c319108-a62c-40fe-a619-0ca069679212-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-279vp\" (UID: \"2c319108-a62c-40fe-a619-0ca069679212\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:05.628817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:05.628798    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:05.749621 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:05.749554    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"]
Jul 06 20:38:05.753138 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:38:05.753108    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c319108_a62c_40fe_a619_0ca069679212.slice/crio-ff2338ea5acbdb0e4f9d6fb885df01d7ce1a8882a122d7301dc3f3e73334e67d WatchSource:0}: Error finding container ff2338ea5acbdb0e4f9d6fb885df01d7ce1a8882a122d7301dc3f3e73334e67d: Status 404 returned error can't find the container with id ff2338ea5acbdb0e4f9d6fb885df01d7ce1a8882a122d7301dc3f3e73334e67d
Jul 06 20:38:06.109456 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.109373    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:38:06.115452 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.115430    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.117465 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.117439    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 06 20:38:06.117581 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.117480    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 06 20:38:06.117581 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.117574    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 06 20:38:06.117965 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.117948    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 06 20:38:06.118354 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118225    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cuckp8appmja4\""
Jul 06 20:38:06.118354 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118238    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 06 20:38:06.118354 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118237    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6bsrk\""
Jul 06 20:38:06.118354 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118311    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 06 20:38:06.118354 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118346    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 06 20:38:06.118645 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118447    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 06 20:38:06.119103 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118943    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 06 20:38:06.119103 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118957    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 06 20:38:06.119103 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118946    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 06 20:38:06.119103 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.118991    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 06 20:38:06.119103 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.119018    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 06 20:38:06.119388 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.119373    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 06 20:38:06.121456 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.121437    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 06 20:38:06.127727 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.127708    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:38:06.292729 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.292695    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.292881 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.292734    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.292881 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.292762    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.292881 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.292792    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.292980 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.292875    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jdb8\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-kube-api-access-8jdb8\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.292980 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.292944    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-config\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.292980 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.292973    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293106 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293002    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293106 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293038    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293188 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293124    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293188 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293142    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293188 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293165    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-web-config\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293289 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293193    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293289 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293212    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293289 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293251    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293289 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293272    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293438 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293327    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-config-out\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293438 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293375    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293438 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293409    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.293438 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.293431    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.394659 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394583    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-config\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.394659 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394621    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.394845 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394776    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.394845 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394816    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.394935 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394850    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.394935 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394873    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.394935 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394904    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-web-config\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395098 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394935    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395098 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394963    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395098 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.394995    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395098 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395018    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395098 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395084    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-config-out\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395136    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395168    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395191    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395229    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395262    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395291    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395317    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.395793 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.395344    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jdb8\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-kube-api-access-8jdb8\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.398172 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.397844    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.399142 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.398340    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.399142 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.398531    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-config\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.399142 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.398537    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.399142 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.398806    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.401353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.399741    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.401353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.399880    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-web-config\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.401353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.399907    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.401353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.400240    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.401353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.400464    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.401353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.400656    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.401353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.400842    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.402094 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.401462    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.402094 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.401699    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-config-out\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.402094 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.401802    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.402376 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.402349    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.403212 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.403187    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.403407 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.403390    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.403626 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.403608    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.403857 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.403843    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jdb8\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-kube-api-access-8jdb8\") pod \"prometheus-k8s-0\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.427924 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.427887    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:06.623830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.623799    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:38:06.627177 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:38:06.627145    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc1bc46_f004_440c_b33c_9a2945084dbd.slice/crio-e252a66f0a7c7626ad4234206cfcf32ad2cd94c26b07e14be42834876c5d04e0 WatchSource:0}: Error finding container e252a66f0a7c7626ad4234206cfcf32ad2cd94c26b07e14be42834876c5d04e0: Status 404 returned error can't find the container with id e252a66f0a7c7626ad4234206cfcf32ad2cd94c26b07e14be42834876c5d04e0
Jul 06 20:38:06.694136 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.694072    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerStarted","Data":"e252a66f0a7c7626ad4234206cfcf32ad2cd94c26b07e14be42834876c5d04e0"}
Jul 06 20:38:06.695420 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:06.695390    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp" event={"ID":"2c319108-a62c-40fe-a619-0ca069679212","Type":"ContainerStarted","Data":"ff2338ea5acbdb0e4f9d6fb885df01d7ce1a8882a122d7301dc3f3e73334e67d"}
Jul 06 20:38:07.707779 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:07.707735    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp" event={"ID":"2c319108-a62c-40fe-a619-0ca069679212","Type":"ContainerStarted","Data":"2c48d0098bbaa0f414dbde0e84fc4ab964b3232d9ebf3cbba427e3d1802f73c8"}
Jul 06 20:38:07.708245 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:07.707918    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:07.714158 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:07.714128    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp"
Jul 06 20:38:07.724873 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:07.724829    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-279vp" podStartSLOduration=2.271335278 podStartE2EDuration="3.724816238s" totalImagesPullingTime="1.45348096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:07.723476414 +0000 UTC m=+163.065379130" watchObservedRunningTime="2026-07-06 20:38:07.724816238 +0000 UTC m=+163.066718956"
Jul 06 20:38:08.512818 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:08.512793    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:38:08.514837 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:08.514818    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22e65afc-1f02-48a2-b7ff-9213eb6cbff7-metrics-tls\") pod \"dns-default-7fbsp\" (UID: \"22e65afc-1f02-48a2-b7ff-9213eb6cbff7\") " pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:38:08.713020 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:08.712996    2541 generic.go:356] "Generic (PLEG): container finished" podID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerID="6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0" exitCode=0
Jul 06 20:38:08.713308 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:08.713083    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerDied","Data":"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"}
Jul 06 20:38:08.783175 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:08.783132    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5jj9\""
Jul 06 20:38:08.792598 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:08.792577    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:38:08.919115 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:08.919036    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7fbsp"]
Jul 06 20:38:08.921567 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:38:08.921540    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e65afc_1f02_48a2_b7ff_9213eb6cbff7.slice/crio-fdce533223eeb7a86f2821fd6801db311978fa3d82d8b1b3c1860660f0b3e38e WatchSource:0}: Error finding container fdce533223eeb7a86f2821fd6801db311978fa3d82d8b1b3c1860660f0b3e38e: Status 404 returned error can't find the container with id fdce533223eeb7a86f2821fd6801db311978fa3d82d8b1b3c1860660f0b3e38e
Jul 06 20:38:09.592795 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.592728    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l" podUID="1c3f951c-98cf-4331-b0da-7370e678e6d2" containerName="registry" containerID="cri-o://620054d25732787a2e8cb56c5dc2c7b467f2417e59b113316f5f3a00caaf2b3b" gracePeriod=30
Jul 06 20:38:09.718569 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.718536    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7fbsp" event={"ID":"22e65afc-1f02-48a2-b7ff-9213eb6cbff7","Type":"ContainerStarted","Data":"fdce533223eeb7a86f2821fd6801db311978fa3d82d8b1b3c1860660f0b3e38e"}
Jul 06 20:38:09.720541 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.720513    2541 generic.go:356] "Generic (PLEG): container finished" podID="1c3f951c-98cf-4331-b0da-7370e678e6d2" containerID="620054d25732787a2e8cb56c5dc2c7b467f2417e59b113316f5f3a00caaf2b3b" exitCode=0
Jul 06 20:38:09.720639 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.720594    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l" event={"ID":"1c3f951c-98cf-4331-b0da-7370e678e6d2","Type":"ContainerDied","Data":"620054d25732787a2e8cb56c5dc2c7b467f2417e59b113316f5f3a00caaf2b3b"}
Jul 06 20:38:09.856401 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.856376    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:38:09.920905 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.920873    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") pod \"1c3f951c-98cf-4331-b0da-7370e678e6d2\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") "
Jul 06 20:38:09.921040 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.920926    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-bound-sa-token\") pod \"1c3f951c-98cf-4331-b0da-7370e678e6d2\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") "
Jul 06 20:38:09.921040 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.920969    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-image-registry-private-configuration\") pod \"1c3f951c-98cf-4331-b0da-7370e678e6d2\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") "
Jul 06 20:38:09.921040 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.921004    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c3f951c-98cf-4331-b0da-7370e678e6d2-ca-trust-extracted\") pod \"1c3f951c-98cf-4331-b0da-7370e678e6d2\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") "
Jul 06 20:38:09.921280 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.921256    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-trusted-ca\") pod \"1c3f951c-98cf-4331-b0da-7370e678e6d2\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") "
Jul 06 20:38:09.921368 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.921302    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-installation-pull-secrets\") pod \"1c3f951c-98cf-4331-b0da-7370e678e6d2\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") "
Jul 06 20:38:09.921368 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.921330    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d9nt\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-kube-api-access-9d9nt\") pod \"1c3f951c-98cf-4331-b0da-7370e678e6d2\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") "
Jul 06 20:38:09.921368 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.921352    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-certificates\") pod \"1c3f951c-98cf-4331-b0da-7370e678e6d2\" (UID: \"1c3f951c-98cf-4331-b0da-7370e678e6d2\") "
Jul 06 20:38:09.922017 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.921987    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1c3f951c-98cf-4331-b0da-7370e678e6d2" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:09.922293 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.922267    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1c3f951c-98cf-4331-b0da-7370e678e6d2" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:38:09.924715 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.924641    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1c3f951c-98cf-4331-b0da-7370e678e6d2" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:09.924715 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.924751    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1c3f951c-98cf-4331-b0da-7370e678e6d2" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:38:09.924955 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.924808    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1c3f951c-98cf-4331-b0da-7370e678e6d2" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:38:09.925888 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.925861    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1c3f951c-98cf-4331-b0da-7370e678e6d2" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:38:09.926462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.926432    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-kube-api-access-9d9nt" (OuterVolumeSpecName: "kube-api-access-9d9nt") pod "1c3f951c-98cf-4331-b0da-7370e678e6d2" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2"). InnerVolumeSpecName "kube-api-access-9d9nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:38:09.932638 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:09.932609    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f951c-98cf-4331-b0da-7370e678e6d2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1c3f951c-98cf-4331-b0da-7370e678e6d2" (UID: "1c3f951c-98cf-4331-b0da-7370e678e6d2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:38:10.022179 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.022146    2541 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-bound-sa-token\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:38:10.022179 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.022179    2541 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-image-registry-private-configuration\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:38:10.022327 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.022193    2541 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1c3f951c-98cf-4331-b0da-7370e678e6d2-ca-trust-extracted\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:38:10.022327 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.022206    2541 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-trusted-ca\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:38:10.022327 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.022221    2541 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1c3f951c-98cf-4331-b0da-7370e678e6d2-installation-pull-secrets\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:38:10.022327 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.022234    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d9nt\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-kube-api-access-9d9nt\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:38:10.022327 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.022246    2541 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-certificates\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:38:10.022327 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.022258    2541 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1c3f951c-98cf-4331-b0da-7370e678e6d2-registry-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:38:10.727530 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.727492    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l" event={"ID":"1c3f951c-98cf-4331-b0da-7370e678e6d2","Type":"ContainerDied","Data":"3ead77fd7fc1ba42c8f66a17a47f258d0c37201bfde69a8c5f36a95c9ff8eaed"}
Jul 06 20:38:10.727960 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.727563    2541 scope.go:117] "RemoveContainer" containerID="620054d25732787a2e8cb56c5dc2c7b467f2417e59b113316f5f3a00caaf2b3b"
Jul 06 20:38:10.727960 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.727712    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c5579b66f-l4r7l"
Jul 06 20:38:10.754026 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.753999    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c5579b66f-l4r7l"]
Jul 06 20:38:10.757516 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:10.757493    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c5579b66f-l4r7l"]
Jul 06 20:38:11.095586 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:11.095541    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3f951c-98cf-4331-b0da-7370e678e6d2" path="/var/lib/kubelet/pods/1c3f951c-98cf-4331-b0da-7370e678e6d2/volumes"
Jul 06 20:38:11.734970 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:11.734931    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7fbsp" event={"ID":"22e65afc-1f02-48a2-b7ff-9213eb6cbff7","Type":"ContainerStarted","Data":"972b3ee26e3f5f2f39a13996e855cae7120c41330b67cba92f0045301e436c71"}
Jul 06 20:38:11.735457 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:11.734998    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7fbsp" event={"ID":"22e65afc-1f02-48a2-b7ff-9213eb6cbff7","Type":"ContainerStarted","Data":"97f8e6903fc30f00f639f85e99922028dcfdce2825a4474eee610e4bad6f5cc5"}
Jul 06 20:38:11.735457 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:11.735101    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:38:11.753398 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:11.753348    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7fbsp" podStartSLOduration=129.820923513 podStartE2EDuration="2m11.753334134s" totalImagesPullingTime="1.932410621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:11.752768501 +0000 UTC m=+167.094671219" watchObservedRunningTime="2026-07-06 20:38:11.753334134 +0000 UTC m=+167.095236850"
Jul 06 20:38:12.741212 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:12.741182    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerStarted","Data":"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"}
Jul 06 20:38:12.741552 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:12.741248    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerStarted","Data":"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"}
Jul 06 20:38:14.756084 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:14.756040    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerStarted","Data":"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"}
Jul 06 20:38:14.756366 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:14.756143    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerStarted","Data":"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"}
Jul 06 20:38:14.756366 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:14.756172    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerStarted","Data":"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"}
Jul 06 20:38:14.756366 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:14.756205    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerStarted","Data":"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"}
Jul 06 20:38:14.786766 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:14.786688    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.091812123 podStartE2EDuration="8.786674726s" totalImagesPullingTime="6.678381476s" totalInitContainerRuntime="16.481127ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:38:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:38:14.785429317 +0000 UTC m=+170.127332036" watchObservedRunningTime="2026-07-06 20:38:14.786674726 +0000 UTC m=+170.128577443"
Jul 06 20:38:16.428079 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:16.427997    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:38:21.743381 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:21.743351    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7fbsp"
Jul 06 20:38:33.829635 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:33.829608    2541 generic.go:356] "Generic (PLEG): container finished" podID="f07831f0-b8dd-4ff7-9808-ab07505f5f3a" containerID="94448f1702cf76a8588add442a28662ee66769bf515c6b3164d8f667bd893cd1" exitCode=0
Jul 06 20:38:33.829946 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:33.829686    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x" event={"ID":"f07831f0-b8dd-4ff7-9808-ab07505f5f3a","Type":"ContainerDied","Data":"94448f1702cf76a8588add442a28662ee66769bf515c6b3164d8f667bd893cd1"}
Jul 06 20:38:33.830019 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:33.830002    2541 scope.go:117] "RemoveContainer" containerID="94448f1702cf76a8588add442a28662ee66769bf515c6b3164d8f667bd893cd1"
Jul 06 20:38:34.835219 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:38:34.835178    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-8wf5x" event={"ID":"f07831f0-b8dd-4ff7-9808-ab07505f5f3a","Type":"ContainerStarted","Data":"8f7b4a0ac3cfeada7d9e400a487f92445156056e0b45a2e1c0e3687c143a7319"}
Jul 06 20:39:06.428198 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:06.428169    2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:06.445893 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:06.445871    2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:06.958297 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:06.958270    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:24.499880 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.499848    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:39:24.500710 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.500659    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="prometheus" containerID="cri-o://3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6" gracePeriod=600
Jul 06 20:39:24.501107 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.501072    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy" containerID="cri-o://0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef" gracePeriod=600
Jul 06 20:39:24.501224 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.501165    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364" gracePeriod=600
Jul 06 20:39:24.501224 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.501212    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy-web" containerID="cri-o://3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22" gracePeriod=600
Jul 06 20:39:24.501336 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.501174    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="thanos-sidecar" containerID="cri-o://841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f" gracePeriod=600
Jul 06 20:39:24.501336 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.501173    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="config-reloader" containerID="cri-o://bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a" gracePeriod=600
Jul 06 20:39:24.745255 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.745234    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:24.824998 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.824918    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-serving-certs-ca-bundle\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.824998 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.824965    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-thanos-prometheus-http-client-file\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.824998 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.824995    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-config-out\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825027    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-metrics-client-ca\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825072    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-1\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825096    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-tls-assets\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825122    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825144    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-metrics-client-certs\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825174    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-web-config\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825235 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825216    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825474 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825246    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-tls\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825474 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825277    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-db\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825474 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825287    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:39:24.825474 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825309    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jdb8\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-kube-api-access-8jdb8\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825474 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825345    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-kube-rbac-proxy\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825474 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825383    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-config\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825474 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825423    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-2\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825474 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825449    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-trusted-ca-bundle\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825847 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825481    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-grpc-tls\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825847 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825507    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-0\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825847 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825534    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-kubelet-serving-ca-bundle\") pod \"3fc1bc46-f004-440c-b33c-9a2945084dbd\" (UID: \"3fc1bc46-f004-440c-b33c-9a2945084dbd\") "
Jul 06 20:39:24.825989 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.825810    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:39:24.828021 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.826344    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:39:24.828021 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.826430    2541 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.828021 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.826451    2541 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-metrics-client-ca\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.828021 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.826466    2541 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.828021 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.826679    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:39:24.828021 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.826961    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:39:24.828021 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.827665    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:39:24.828780 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.828677    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:39:24.828780 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.828746    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:39:24.828957 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.828936    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:39:24.829456 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.829423    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:39:24.830528 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.830504    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-config-out" (OuterVolumeSpecName: "config-out") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:39:24.830528 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.830544    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:39:24.830528 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.830686    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:39:24.830528 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.830865    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:39:24.830528 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.830956    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-config" (OuterVolumeSpecName: "config") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:39:24.831679 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.831660    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:39:24.831837 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.831811    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-kube-api-access-8jdb8" (OuterVolumeSpecName: "kube-api-access-8jdb8") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "kube-api-access-8jdb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:39:24.831837 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.831824    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:39:24.832240 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.832219    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:39:24.841401 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.841380    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-web-config" (OuterVolumeSpecName: "web-config") pod "3fc1bc46-f004-440c-b33c-9a2945084dbd" (UID: "3fc1bc46-f004-440c-b33c-9a2945084dbd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:39:24.926933 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926910    2541 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-kube-rbac-proxy\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.926933 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926930    2541 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926939    2541 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926948    2541 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926957    2541 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-grpc-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926965    2541 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926974    2541 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fc1bc46-f004-440c-b33c-9a2945084dbd-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926982    2541 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926990    2541 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-config-out\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.926997    2541 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-tls-assets\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.927004    2541 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.927011    2541 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-metrics-client-certs\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.927018    2541 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-web-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.927026    2541 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.927033    2541 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3fc1bc46-f004-440c-b33c-9a2945084dbd-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.927041    2541 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3fc1bc46-f004-440c-b33c-9a2945084dbd-prometheus-k8s-db\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:24.927041 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:24.927062    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jdb8\" (UniqueName: \"kubernetes.io/projected/3fc1bc46-f004-440c-b33c-9a2945084dbd-kube-api-access-8jdb8\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:39:25.008428 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008400    2541 generic.go:356] "Generic (PLEG): container finished" podID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerID="bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364" exitCode=0
Jul 06 20:39:25.008428 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008427    2541 generic.go:356] "Generic (PLEG): container finished" podID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerID="0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef" exitCode=0
Jul 06 20:39:25.008562 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008436    2541 generic.go:356] "Generic (PLEG): container finished" podID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerID="3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22" exitCode=0
Jul 06 20:39:25.008562 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008444    2541 generic.go:356] "Generic (PLEG): container finished" podID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerID="841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f" exitCode=0
Jul 06 20:39:25.008562 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008451    2541 generic.go:356] "Generic (PLEG): container finished" podID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerID="bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a" exitCode=0
Jul 06 20:39:25.008562 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008457    2541 generic.go:356] "Generic (PLEG): container finished" podID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerID="3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6" exitCode=0
Jul 06 20:39:25.008562 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008474    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerDied","Data":"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"}
Jul 06 20:39:25.008562 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008533    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.008562 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008553    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerDied","Data":"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"}
Jul 06 20:39:25.008812 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008578    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerDied","Data":"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"}
Jul 06 20:39:25.008812 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008603    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerDied","Data":"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"}
Jul 06 20:39:25.008812 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008626    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerDied","Data":"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"}
Jul 06 20:39:25.008812 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008628    2541 scope.go:117] "RemoveContainer" containerID="bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"
Jul 06 20:39:25.008812 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008647    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerDied","Data":"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"}
Jul 06 20:39:25.008812 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.008670    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fc1bc46-f004-440c-b33c-9a2945084dbd","Type":"ContainerDied","Data":"e252a66f0a7c7626ad4234206cfcf32ad2cd94c26b07e14be42834876c5d04e0"}
Jul 06 20:39:25.015949 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.015921    2541 scope.go:117] "RemoveContainer" containerID="0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"
Jul 06 20:39:25.024651 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.024635    2541 scope.go:117] "RemoveContainer" containerID="3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"
Jul 06 20:39:25.030717 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.030701    2541 scope.go:117] "RemoveContainer" containerID="841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"
Jul 06 20:39:25.033183 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.033162    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:39:25.037159 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.037132    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:39:25.037815 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.037801    2541 scope.go:117] "RemoveContainer" containerID="bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"
Jul 06 20:39:25.044125 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.044109    2541 scope.go:117] "RemoveContainer" containerID="3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"
Jul 06 20:39:25.051541 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.051526    2541 scope.go:117] "RemoveContainer" containerID="6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"
Jul 06 20:39:25.057628 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.057613    2541 scope.go:117] "RemoveContainer" containerID="bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"
Jul 06 20:39:25.057879 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:39:25.057852    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": container with ID starting with bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364 not found: ID does not exist" containerID="bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"
Jul 06 20:39:25.057942 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.057886    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"} err="failed to get container status \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": rpc error: code = NotFound desc = could not find container \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": container with ID starting with bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364 not found: ID does not exist"
Jul 06 20:39:25.057942 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.057911    2541 scope.go:117] "RemoveContainer" containerID="0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"
Jul 06 20:39:25.058283 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:39:25.058265    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": container with ID starting with 0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef not found: ID does not exist" containerID="0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"
Jul 06 20:39:25.058338 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.058289    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"} err="failed to get container status \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": rpc error: code = NotFound desc = could not find container \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": container with ID starting with 0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef not found: ID does not exist"
Jul 06 20:39:25.058338 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.058303    2541 scope.go:117] "RemoveContainer" containerID="3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"
Jul 06 20:39:25.058536 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:39:25.058520    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": container with ID starting with 3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22 not found: ID does not exist" containerID="3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"
Jul 06 20:39:25.058574 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.058541    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"} err="failed to get container status \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": rpc error: code = NotFound desc = could not find container \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": container with ID starting with 3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22 not found: ID does not exist"
Jul 06 20:39:25.058574 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.058554    2541 scope.go:117] "RemoveContainer" containerID="841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"
Jul 06 20:39:25.058751 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:39:25.058736    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": container with ID starting with 841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f not found: ID does not exist" containerID="841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"
Jul 06 20:39:25.058787 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.058757    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"} err="failed to get container status \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": rpc error: code = NotFound desc = could not find container \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": container with ID starting with 841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f not found: ID does not exist"
Jul 06 20:39:25.058787 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.058770    2541 scope.go:117] "RemoveContainer" containerID="bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"
Jul 06 20:39:25.058957 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:39:25.058938    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": container with ID starting with bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a not found: ID does not exist" containerID="bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"
Jul 06 20:39:25.058996 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.058963    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"} err="failed to get container status \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": rpc error: code = NotFound desc = could not find container \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": container with ID starting with bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a not found: ID does not exist"
Jul 06 20:39:25.058996 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.058975    2541 scope.go:117] "RemoveContainer" containerID="3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"
Jul 06 20:39:25.059200 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:39:25.059183    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": container with ID starting with 3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6 not found: ID does not exist" containerID="3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"
Jul 06 20:39:25.059266 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.059206    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"} err="failed to get container status \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": rpc error: code = NotFound desc = could not find container \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": container with ID starting with 3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6 not found: ID does not exist"
Jul 06 20:39:25.059266 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.059225    2541 scope.go:117] "RemoveContainer" containerID="6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"
Jul 06 20:39:25.059435 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:39:25.059421    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": container with ID starting with 6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0 not found: ID does not exist" containerID="6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"
Jul 06 20:39:25.059467 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.059439    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"} err="failed to get container status \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": rpc error: code = NotFound desc = could not find container \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": container with ID starting with 6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0 not found: ID does not exist"
Jul 06 20:39:25.059467 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.059451    2541 scope.go:117] "RemoveContainer" containerID="bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"
Jul 06 20:39:25.059671 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.059654    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"} err="failed to get container status \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": rpc error: code = NotFound desc = could not find container \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": container with ID starting with bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364 not found: ID does not exist"
Jul 06 20:39:25.059739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.059672    2541 scope.go:117] "RemoveContainer" containerID="0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"
Jul 06 20:39:25.059873 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.059858    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"} err="failed to get container status \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": rpc error: code = NotFound desc = could not find container \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": container with ID starting with 0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef not found: ID does not exist"
Jul 06 20:39:25.059915 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.059875    2541 scope.go:117] "RemoveContainer" containerID="3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"
Jul 06 20:39:25.060039 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060023    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"} err="failed to get container status \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": rpc error: code = NotFound desc = could not find container \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": container with ID starting with 3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22 not found: ID does not exist"
Jul 06 20:39:25.060039 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060040    2541 scope.go:117] "RemoveContainer" containerID="841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"
Jul 06 20:39:25.060264 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060248    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"} err="failed to get container status \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": rpc error: code = NotFound desc = could not find container \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": container with ID starting with 841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f not found: ID does not exist"
Jul 06 20:39:25.060299 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060263    2541 scope.go:117] "RemoveContainer" containerID="bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"
Jul 06 20:39:25.060438 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060425    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"} err="failed to get container status \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": rpc error: code = NotFound desc = could not find container \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": container with ID starting with bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a not found: ID does not exist"
Jul 06 20:39:25.060438 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060437    2541 scope.go:117] "RemoveContainer" containerID="3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"
Jul 06 20:39:25.060601 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060588    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"} err="failed to get container status \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": rpc error: code = NotFound desc = could not find container \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": container with ID starting with 3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6 not found: ID does not exist"
Jul 06 20:39:25.060601 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060600    2541 scope.go:117] "RemoveContainer" containerID="6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"
Jul 06 20:39:25.060767 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060751    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"} err="failed to get container status \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": rpc error: code = NotFound desc = could not find container \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": container with ID starting with 6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0 not found: ID does not exist"
Jul 06 20:39:25.060807 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060767    2541 scope.go:117] "RemoveContainer" containerID="bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"
Jul 06 20:39:25.060940 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060926    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"} err="failed to get container status \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": rpc error: code = NotFound desc = could not find container \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": container with ID starting with bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364 not found: ID does not exist"
Jul 06 20:39:25.060980 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.060939    2541 scope.go:117] "RemoveContainer" containerID="0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"
Jul 06 20:39:25.061112 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061097    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"} err="failed to get container status \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": rpc error: code = NotFound desc = could not find container \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": container with ID starting with 0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef not found: ID does not exist"
Jul 06 20:39:25.061175 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061113    2541 scope.go:117] "RemoveContainer" containerID="3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"
Jul 06 20:39:25.061330 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061315    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"} err="failed to get container status \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": rpc error: code = NotFound desc = could not find container \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": container with ID starting with 3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22 not found: ID does not exist"
Jul 06 20:39:25.061378 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061331    2541 scope.go:117] "RemoveContainer" containerID="841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"
Jul 06 20:39:25.061497 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061481    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"} err="failed to get container status \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": rpc error: code = NotFound desc = could not find container \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": container with ID starting with 841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f not found: ID does not exist"
Jul 06 20:39:25.061545 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061498    2541 scope.go:117] "RemoveContainer" containerID="bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"
Jul 06 20:39:25.061666 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061652    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"} err="failed to get container status \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": rpc error: code = NotFound desc = could not find container \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": container with ID starting with bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a not found: ID does not exist"
Jul 06 20:39:25.061712 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061667    2541 scope.go:117] "RemoveContainer" containerID="3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"
Jul 06 20:39:25.061847 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061833    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"} err="failed to get container status \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": rpc error: code = NotFound desc = could not find container \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": container with ID starting with 3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6 not found: ID does not exist"
Jul 06 20:39:25.061847 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061847    2541 scope.go:117] "RemoveContainer" containerID="6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"
Jul 06 20:39:25.062012 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.061996    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"} err="failed to get container status \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": rpc error: code = NotFound desc = could not find container \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": container with ID starting with 6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0 not found: ID does not exist"
Jul 06 20:39:25.062068 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.062011    2541 scope.go:117] "RemoveContainer" containerID="bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"
Jul 06 20:39:25.062204 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.062189    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"} err="failed to get container status \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": rpc error: code = NotFound desc = could not find container \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": container with ID starting with bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364 not found: ID does not exist"
Jul 06 20:39:25.062283 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.062204    2541 scope.go:117] "RemoveContainer" containerID="0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"
Jul 06 20:39:25.062385 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.062371    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"} err="failed to get container status \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": rpc error: code = NotFound desc = could not find container \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": container with ID starting with 0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef not found: ID does not exist"
Jul 06 20:39:25.062428 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.062385    2541 scope.go:117] "RemoveContainer" containerID="3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"
Jul 06 20:39:25.062550 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.062534    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"} err="failed to get container status \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": rpc error: code = NotFound desc = could not find container \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": container with ID starting with 3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22 not found: ID does not exist"
Jul 06 20:39:25.062600 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.062551    2541 scope.go:117] "RemoveContainer" containerID="841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"
Jul 06 20:39:25.062739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.062714    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"} err="failed to get container status \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": rpc error: code = NotFound desc = could not find container \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": container with ID starting with 841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f not found: ID does not exist"
Jul 06 20:39:25.062739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.062739    2541 scope.go:117] "RemoveContainer" containerID="bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"
Jul 06 20:39:25.063131 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.063113    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"} err="failed to get container status \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": rpc error: code = NotFound desc = could not find container \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": container with ID starting with bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a not found: ID does not exist"
Jul 06 20:39:25.063313 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.063136    2541 scope.go:117] "RemoveContainer" containerID="3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"
Jul 06 20:39:25.063503 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.063485    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"} err="failed to get container status \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": rpc error: code = NotFound desc = could not find container \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": container with ID starting with 3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6 not found: ID does not exist"
Jul 06 20:39:25.063594 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.063504    2541 scope.go:117] "RemoveContainer" containerID="6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"
Jul 06 20:39:25.063761 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.063737    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"} err="failed to get container status \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": rpc error: code = NotFound desc = could not find container \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": container with ID starting with 6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0 not found: ID does not exist"
Jul 06 20:39:25.063761 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.063759    2541 scope.go:117] "RemoveContainer" containerID="bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"
Jul 06 20:39:25.063984 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.063964    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"} err="failed to get container status \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": rpc error: code = NotFound desc = could not find container \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": container with ID starting with bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364 not found: ID does not exist"
Jul 06 20:39:25.064024 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.063986    2541 scope.go:117] "RemoveContainer" containerID="0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"
Jul 06 20:39:25.064241 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064223    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"} err="failed to get container status \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": rpc error: code = NotFound desc = could not find container \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": container with ID starting with 0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef not found: ID does not exist"
Jul 06 20:39:25.064343 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064243    2541 scope.go:117] "RemoveContainer" containerID="3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"
Jul 06 20:39:25.064343 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064246    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:39:25.064477 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064461    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"} err="failed to get container status \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": rpc error: code = NotFound desc = could not find container \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": container with ID starting with 3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22 not found: ID does not exist"
Jul 06 20:39:25.064518 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064478    2541 scope.go:117] "RemoveContainer" containerID="841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"
Jul 06 20:39:25.064518 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064493    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy-web"
Jul 06 20:39:25.064518 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064506    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy-web"
Jul 06 20:39:25.064518 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064515    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy-thanos"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064520    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy-thanos"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064532    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3f951c-98cf-4331-b0da-7370e678e6d2" containerName="registry"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064537    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3f951c-98cf-4331-b0da-7370e678e6d2" containerName="registry"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064544    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064549    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064557    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="thanos-sidecar"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064562    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="thanos-sidecar"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064577    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="init-config-reloader"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064584    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="init-config-reloader"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064595    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="prometheus"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064603    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="prometheus"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064614    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="config-reloader"
Jul 06 20:39:25.064655 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064621    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="config-reloader"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064676    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy-web"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064691    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy-thanos"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064696    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="prometheus"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064701    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="thanos-sidecar"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064705    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="config-reloader"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064709    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" containerName="kube-rbac-proxy"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064713    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3f951c-98cf-4331-b0da-7370e678e6d2" containerName="registry"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064671    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"} err="failed to get container status \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": rpc error: code = NotFound desc = could not find container \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": container with ID starting with 841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f not found: ID does not exist"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064759    2541 scope.go:117] "RemoveContainer" containerID="bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064974    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"} err="failed to get container status \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": rpc error: code = NotFound desc = could not find container \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": container with ID starting with bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a not found: ID does not exist"
Jul 06 20:39:25.065016 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.064990    2541 scope.go:117] "RemoveContainer" containerID="3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"
Jul 06 20:39:25.065430 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065195    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"} err="failed to get container status \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": rpc error: code = NotFound desc = could not find container \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": container with ID starting with 3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6 not found: ID does not exist"
Jul 06 20:39:25.065430 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065219    2541 scope.go:117] "RemoveContainer" containerID="6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"
Jul 06 20:39:25.065430 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065397    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"} err="failed to get container status \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": rpc error: code = NotFound desc = could not find container \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": container with ID starting with 6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0 not found: ID does not exist"
Jul 06 20:39:25.065430 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065411    2541 scope.go:117] "RemoveContainer" containerID="bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"
Jul 06 20:39:25.065605 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065583    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364"} err="failed to get container status \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": rpc error: code = NotFound desc = could not find container \"bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364\": container with ID starting with bf3a1dc41d03cefb184d6771de35dbddf7bedfeaaa6c94838fa5809d3fb39364 not found: ID does not exist"
Jul 06 20:39:25.065645 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065606    2541 scope.go:117] "RemoveContainer" containerID="0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"
Jul 06 20:39:25.065789 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065772    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef"} err="failed to get container status \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": rpc error: code = NotFound desc = could not find container \"0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef\": container with ID starting with 0dc7d64bd5fe2dc79c9b90928c6f7232b5dead948eb2f9bc3ca09359033221ef not found: ID does not exist"
Jul 06 20:39:25.065830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065789    2541 scope.go:117] "RemoveContainer" containerID="3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"
Jul 06 20:39:25.065981 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065966    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22"} err="failed to get container status \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": rpc error: code = NotFound desc = could not find container \"3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22\": container with ID starting with 3e048c7f52d41da9dced1232069ef8b494f51a31c5658b062f29fb993817ba22 not found: ID does not exist"
Jul 06 20:39:25.066019 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.065981    2541 scope.go:117] "RemoveContainer" containerID="841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"
Jul 06 20:39:25.066187 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.066170    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f"} err="failed to get container status \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": rpc error: code = NotFound desc = could not find container \"841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f\": container with ID starting with 841f5a85bbafb84b80a3b94582c67d57186a07cfa925c82f277b85c6c804e43f not found: ID does not exist"
Jul 06 20:39:25.066286 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.066189    2541 scope.go:117] "RemoveContainer" containerID="bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"
Jul 06 20:39:25.066435 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.066417    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a"} err="failed to get container status \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": rpc error: code = NotFound desc = could not find container \"bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a\": container with ID starting with bb53c18e0b036638f4ca506ef50af800581237e435d4d5dea1d17d9e7b52a74a not found: ID does not exist"
Jul 06 20:39:25.066481 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.066435    2541 scope.go:117] "RemoveContainer" containerID="3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"
Jul 06 20:39:25.066609 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.066594    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6"} err="failed to get container status \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": rpc error: code = NotFound desc = could not find container \"3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6\": container with ID starting with 3433262c6e2fd270346b1d09dacfd9c6cf5bb1e881edf8a5ea5fc15ccb3b49e6 not found: ID does not exist"
Jul 06 20:39:25.066649 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.066608    2541 scope.go:117] "RemoveContainer" containerID="6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"
Jul 06 20:39:25.066772 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.066758    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0"} err="failed to get container status \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": rpc error: code = NotFound desc = could not find container \"6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0\": container with ID starting with 6f73e93ac75a8ca4f680f6479890177c38d83b62098403e1e4e249639d7e92d0 not found: ID does not exist"
Jul 06 20:39:25.070234 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.070212    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.072258 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.072242    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 06 20:39:25.072524 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.072505    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 06 20:39:25.072613 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.072525    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cuckp8appmja4\""
Jul 06 20:39:25.072613 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.072525    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 06 20:39:25.072794 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.072780    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 06 20:39:25.072853 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.072838    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 06 20:39:25.073230 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.073215    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 06 20:39:25.073314 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.073253    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6bsrk\""
Jul 06 20:39:25.073604 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.073437    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 06 20:39:25.073604 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.073456    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 06 20:39:25.073872 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.073776    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 06 20:39:25.073872 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.073815    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 06 20:39:25.074340 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.074320    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 06 20:39:25.074429 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.074339    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 06 20:39:25.074819 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.074788    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 06 20:39:25.075706 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.075658    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 06 20:39:25.079384 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.079366    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 06 20:39:25.081955 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.081937    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:39:25.094504 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.094484    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fc1bc46-f004-440c-b33c-9a2945084dbd" path="/var/lib/kubelet/pods/3fc1bc46-f004-440c-b33c-9a2945084dbd/volumes"
Jul 06 20:39:25.127996 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.127976    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbq8r\" (UniqueName: \"kubernetes.io/projected/c63b9af9-5191-4b42-af74-7104806be410-kube-api-access-xbq8r\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128112 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128003    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128112 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128021    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128112 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128038    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128112 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128080    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128129    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c63b9af9-5191-4b42-af74-7104806be410-config-out\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128181    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128206    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128253    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-config\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128280    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128305    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-web-config\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128326    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128345    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128362    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128376    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c63b9af9-5191-4b42-af74-7104806be410-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128392    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128614 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128412    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128614 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128427    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128614 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128442    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.128614 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.128456    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229025 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229001    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229151 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229037    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-web-config\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229151 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229088    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229151 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229114    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229151 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229142    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229324 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229165    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c63b9af9-5191-4b42-af74-7104806be410-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229324 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229210    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229324 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229245    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229324 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229273    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229324 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229294    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229324 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229314    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229343    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbq8r\" (UniqueName: \"kubernetes.io/projected/c63b9af9-5191-4b42-af74-7104806be410-kube-api-access-xbq8r\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229371    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229413    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229441    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229466    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229494    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c63b9af9-5191-4b42-af74-7104806be410-config-out\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229519    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229564    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.229610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229571    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.230169 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229626    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-config\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.230169 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229738    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.230169 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.229766    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.230430 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.230388    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.230981 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.230715    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.230981 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.230945    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.231146 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.231036    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.232506 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.232482    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-web-config\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.233085 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.232993    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.233085 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.232995    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c63b9af9-5191-4b42-af74-7104806be410-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.233245 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.233124    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.233824 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.233802    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.233919 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.233853    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.234067 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.234030    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c63b9af9-5191-4b42-af74-7104806be410-config-out\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.234736 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.234710    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c63b9af9-5191-4b42-af74-7104806be410-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.234829 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.234719    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-config\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.235067 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.235027    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.235455 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.235435    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.235524 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.235512    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c63b9af9-5191-4b42-af74-7104806be410-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.238488 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.238470    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbq8r\" (UniqueName: \"kubernetes.io/projected/c63b9af9-5191-4b42-af74-7104806be410-kube-api-access-xbq8r\") pod \"prometheus-k8s-0\" (UID: \"c63b9af9-5191-4b42-af74-7104806be410\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.381558 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.381517    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:39:25.523868 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:25.523844    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 06 20:39:25.525768 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:39:25.525746    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63b9af9_5191_4b42_af74_7104806be410.slice/crio-e9af7df03f316dfcc974261181ccdd09ffe1319a1c62f3f92c3e0ee81a9f591c WatchSource:0}: Error finding container e9af7df03f316dfcc974261181ccdd09ffe1319a1c62f3f92c3e0ee81a9f591c: Status 404 returned error can't find the container with id e9af7df03f316dfcc974261181ccdd09ffe1319a1c62f3f92c3e0ee81a9f591c
Jul 06 20:39:26.014439 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:26.014409    2541 generic.go:356] "Generic (PLEG): container finished" podID="c63b9af9-5191-4b42-af74-7104806be410" containerID="ffcfde6c37ab6f33b1a71ed4898d541916ebb537b15fa2393b349917604e1aca" exitCode=0
Jul 06 20:39:26.014536 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:26.014485    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c63b9af9-5191-4b42-af74-7104806be410","Type":"ContainerDied","Data":"ffcfde6c37ab6f33b1a71ed4898d541916ebb537b15fa2393b349917604e1aca"}
Jul 06 20:39:26.014610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:26.014556    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c63b9af9-5191-4b42-af74-7104806be410","Type":"ContainerStarted","Data":"e9af7df03f316dfcc974261181ccdd09ffe1319a1c62f3f92c3e0ee81a9f591c"}
Jul 06 20:39:27.022111 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:27.022080    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c63b9af9-5191-4b42-af74-7104806be410","Type":"ContainerStarted","Data":"2439a724700a487d5c87440bc2900c75f7ba1c6845b3667be82da1fbb9925ce4"}
Jul 06 20:39:27.022500 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:27.022193    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c63b9af9-5191-4b42-af74-7104806be410","Type":"ContainerStarted","Data":"497561efcb0bc3f2ff78e141f602a96ad1560c249585745778a743f2834e7285"}
Jul 06 20:39:27.022500 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:27.022263    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c63b9af9-5191-4b42-af74-7104806be410","Type":"ContainerStarted","Data":"7d39230ee920cd0c738626f08b9e29374fa7143698797e3409dec57e99fd2bb8"}
Jul 06 20:39:27.022500 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:27.022308    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c63b9af9-5191-4b42-af74-7104806be410","Type":"ContainerStarted","Data":"c58e8ce40267195b4f6f885f878cac3b9f0905661f09f687447a556895ad6f43"}
Jul 06 20:39:27.022500 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:27.022348    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c63b9af9-5191-4b42-af74-7104806be410","Type":"ContainerStarted","Data":"d14e5056afe31956009e990b8f911ada994455f6f65414360d5b7a668bdf7764"}
Jul 06 20:39:27.022500 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:27.022390    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c63b9af9-5191-4b42-af74-7104806be410","Type":"ContainerStarted","Data":"8781911865fe98b5a6eb32e23b1824c9b77e4075cb993fbde6104661ba18a491"}
Jul 06 20:39:27.050748 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:27.050703    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.032356215 podStartE2EDuration="2.050688857s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.332642ms" isStatefulPod=false podCreationTimestamp="2026-07-06 20:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:39:27.049017253 +0000 UTC m=+242.390919969" watchObservedRunningTime="2026-07-06 20:39:27.050688857 +0000 UTC m=+242.392591574"
Jul 06 20:39:30.382604 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:39:30.382566    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:40:24.975513 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:40:24.975476    2541 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 20:40:25.381908 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:40:25.381859    2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:40:25.396537 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:40:25.396518    2541 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:40:26.244112 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:40:26.244091    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 06 20:44:14.564006 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.563974    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-nv9cx"]
Jul 06 20:44:14.567127 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.567111    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:44:14.569312 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.569290    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 06 20:44:14.569312 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.569308    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 06 20:44:14.569695 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.569682    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-x6b47\""
Jul 06 20:44:14.569742 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.569705    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 06 20:44:14.572932 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.572907    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nv9cx"]
Jul 06 20:44:14.632280 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.632259    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn7h5\" (UniqueName: \"kubernetes.io/projected/b5de7e6a-8427-4b18-a51d-669e3ba61fee-kube-api-access-nn7h5\") pod \"seaweedfs-64568bcd49-nv9cx\" (UID: \"b5de7e6a-8427-4b18-a51d-669e3ba61fee\") " pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:44:14.632372 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.632308    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b5de7e6a-8427-4b18-a51d-669e3ba61fee-data\") pod \"seaweedfs-64568bcd49-nv9cx\" (UID: \"b5de7e6a-8427-4b18-a51d-669e3ba61fee\") " pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:44:14.733065 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.733022    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b5de7e6a-8427-4b18-a51d-669e3ba61fee-data\") pod \"seaweedfs-64568bcd49-nv9cx\" (UID: \"b5de7e6a-8427-4b18-a51d-669e3ba61fee\") " pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:44:14.733171 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.733094    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn7h5\" (UniqueName: \"kubernetes.io/projected/b5de7e6a-8427-4b18-a51d-669e3ba61fee-kube-api-access-nn7h5\") pod \"seaweedfs-64568bcd49-nv9cx\" (UID: \"b5de7e6a-8427-4b18-a51d-669e3ba61fee\") " pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:44:14.733342 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.733325    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b5de7e6a-8427-4b18-a51d-669e3ba61fee-data\") pod \"seaweedfs-64568bcd49-nv9cx\" (UID: \"b5de7e6a-8427-4b18-a51d-669e3ba61fee\") " pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:44:14.740273 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.740254    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn7h5\" (UniqueName: \"kubernetes.io/projected/b5de7e6a-8427-4b18-a51d-669e3ba61fee-kube-api-access-nn7h5\") pod \"seaweedfs-64568bcd49-nv9cx\" (UID: \"b5de7e6a-8427-4b18-a51d-669e3ba61fee\") " pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:44:14.877362 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.877295    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:44:14.999758 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:14.999733    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nv9cx"]
Jul 06 20:44:15.002504 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:44:15.002478    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5de7e6a_8427_4b18_a51d_669e3ba61fee.slice/crio-b916576f2a7daa132fa581e19d89498cf80fff6baa1eea40144b428352d0d204 WatchSource:0}: Error finding container b916576f2a7daa132fa581e19d89498cf80fff6baa1eea40144b428352d0d204: Status 404 returned error can't find the container with id b916576f2a7daa132fa581e19d89498cf80fff6baa1eea40144b428352d0d204
Jul 06 20:44:15.003832 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:15.003814    2541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:44:16.003817 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:16.003777    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nv9cx" event={"ID":"b5de7e6a-8427-4b18-a51d-669e3ba61fee","Type":"ContainerStarted","Data":"b916576f2a7daa132fa581e19d89498cf80fff6baa1eea40144b428352d0d204"}
Jul 06 20:44:19.017394 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:19.017365    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nv9cx" event={"ID":"b5de7e6a-8427-4b18-a51d-669e3ba61fee","Type":"ContainerStarted","Data":"2fe33c90faf11a6c92f325a12aa2d5926e0e49934617fd77abf7b580cf2585c1"}
Jul 06 20:44:19.017739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:19.017490    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:44:19.032493 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:19.032447    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-nv9cx" podStartSLOduration=1.380516159 podStartE2EDuration="5.032431387s" totalImagesPullingTime="3.651915228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:44:19.031727131 +0000 UTC m=+534.373629846" watchObservedRunningTime="2026-07-06 20:44:19.032431387 +0000 UTC m=+534.374334103"
Jul 06 20:44:25.023402 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:44:25.023370    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-nv9cx"
Jul 06 20:45:23.195381 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.195350    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-mhmm6"]
Jul 06 20:45:23.198611 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.198592    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:23.201069 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.200942    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-wzw7v\""
Jul 06 20:45:23.205427 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.205403    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 06 20:45:23.208920 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.208900    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-mhmm6"]
Jul 06 20:45:23.294865 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.294843    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xtb4\" (UniqueName: \"kubernetes.io/projected/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd-kube-api-access-7xtb4\") pod \"odh-model-controller-84b54b7f97-mhmm6\" (UID: \"aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd\") " pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:23.294996 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.294872    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd-cert\") pod \"odh-model-controller-84b54b7f97-mhmm6\" (UID: \"aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd\") " pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:23.395351 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.395324    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xtb4\" (UniqueName: \"kubernetes.io/projected/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd-kube-api-access-7xtb4\") pod \"odh-model-controller-84b54b7f97-mhmm6\" (UID: \"aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd\") " pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:23.395351 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.395352    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd-cert\") pod \"odh-model-controller-84b54b7f97-mhmm6\" (UID: \"aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd\") " pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:23.397495 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.397471    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd-cert\") pod \"odh-model-controller-84b54b7f97-mhmm6\" (UID: \"aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd\") " pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:23.401966 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.401944    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xtb4\" (UniqueName: \"kubernetes.io/projected/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd-kube-api-access-7xtb4\") pod \"odh-model-controller-84b54b7f97-mhmm6\" (UID: \"aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd\") " pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:23.511977 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:23.511946    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:26.097343 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:45:26.097312    2541 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 06 20:45:26.097343 ip-10-0-134-39 kubenswrapper[2541]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-84b54b7f97-mhmm6_kserve_aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd_0(0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3): error adding pod kserve_odh-model-controller-84b54b7f97-mhmm6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3" Netns:"/var/run/netns/00c842f3-1014-4c0a-bd8d-ee1eaec744ab" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kserve;K8S_POD_NAME=odh-model-controller-84b54b7f97-mhmm6;K8S_POD_INFRA_CONTAINER_ID=0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3;K8S_POD_UID=aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd" Path:"" ERRORED: error configuring pod [kserve/odh-model-controller-84b54b7f97-mhmm6] networking: Multus: [kserve/odh-model-controller-84b54b7f97-mhmm6/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd]: error waiting for pod: Get "https://a02a453c93518445492c708b3b388bf1-2230622eb921ce79.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kserve/pods/odh-model-controller-84b54b7f97-mhmm6?timeout=1m0s": context deadline exceeded
Jul 06 20:45:26.097343 ip-10-0-134-39 kubenswrapper[2541]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 06 20:45:26.097343 ip-10-0-134-39 kubenswrapper[2541]:  >
Jul 06 20:45:26.097788 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:45:26.097359    2541 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 06 20:45:26.097788 ip-10-0-134-39 kubenswrapper[2541]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-84b54b7f97-mhmm6_kserve_aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd_0(0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3): error adding pod kserve_odh-model-controller-84b54b7f97-mhmm6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3" Netns:"/var/run/netns/00c842f3-1014-4c0a-bd8d-ee1eaec744ab" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kserve;K8S_POD_NAME=odh-model-controller-84b54b7f97-mhmm6;K8S_POD_INFRA_CONTAINER_ID=0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3;K8S_POD_UID=aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd" Path:"" ERRORED: error configuring pod [kserve/odh-model-controller-84b54b7f97-mhmm6] networking: Multus: [kserve/odh-model-controller-84b54b7f97-mhmm6/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd]: error waiting for pod: Get "https://a02a453c93518445492c708b3b388bf1-2230622eb921ce79.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kserve/pods/odh-model-controller-84b54b7f97-mhmm6?timeout=1m0s": context deadline exceeded
Jul 06 20:45:26.097788 ip-10-0-134-39 kubenswrapper[2541]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 06 20:45:26.097788 ip-10-0-134-39 kubenswrapper[2541]:  > pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:26.097788 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:45:26.097374    2541 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 06 20:45:26.097788 ip-10-0-134-39 kubenswrapper[2541]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-84b54b7f97-mhmm6_kserve_aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd_0(0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3): error adding pod kserve_odh-model-controller-84b54b7f97-mhmm6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3" Netns:"/var/run/netns/00c842f3-1014-4c0a-bd8d-ee1eaec744ab" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kserve;K8S_POD_NAME=odh-model-controller-84b54b7f97-mhmm6;K8S_POD_INFRA_CONTAINER_ID=0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3;K8S_POD_UID=aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd" Path:"" ERRORED: error configuring pod [kserve/odh-model-controller-84b54b7f97-mhmm6] networking: Multus: [kserve/odh-model-controller-84b54b7f97-mhmm6/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd]: error waiting for pod: Get "https://a02a453c93518445492c708b3b388bf1-2230622eb921ce79.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kserve/pods/odh-model-controller-84b54b7f97-mhmm6?timeout=1m0s": context deadline exceeded
Jul 06 20:45:26.097788 ip-10-0-134-39 kubenswrapper[2541]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 06 20:45:26.097788 ip-10-0-134-39 kubenswrapper[2541]:  > pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:26.097788 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:45:26.097429    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"odh-model-controller-84b54b7f97-mhmm6_kserve(aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"odh-model-controller-84b54b7f97-mhmm6_kserve(aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-84b54b7f97-mhmm6_kserve_aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd_0(0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3): error adding pod kserve_odh-model-controller-84b54b7f97-mhmm6 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3\\\" Netns:\\\"/var/run/netns/00c842f3-1014-4c0a-bd8d-ee1eaec744ab\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=kserve;K8S_POD_NAME=odh-model-controller-84b54b7f97-mhmm6;K8S_POD_INFRA_CONTAINER_ID=0f295bd49a61afd918f1a3149a59f59f9d299e90ee52504c6cee4574b26200b3;K8S_POD_UID=aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd\\\" Path:\\\"\\\" ERRORED: error configuring pod [kserve/odh-model-controller-84b54b7f97-mhmm6] networking: Multus: [kserve/odh-model-controller-84b54b7f97-mhmm6/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd]: error waiting for pod: Get \\\"https://a02a453c93518445492c708b3b388bf1-2230622eb921ce79.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kserve/pods/odh-model-controller-84b54b7f97-mhmm6?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="kserve/odh-model-controller-84b54b7f97-mhmm6" podUID="aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd"
Jul 06 20:45:26.246949 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:26.246925    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:26.247161 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:26.247149    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:28.833822 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:45:28.833795    2541 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 06 20:45:28.833822 ip-10-0-134-39 kubenswrapper[2541]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-84b54b7f97-mhmm6_kserve_aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd_0(fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf): error adding pod kserve_odh-model-controller-84b54b7f97-mhmm6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf" Netns:"/var/run/netns/8101b75d-944e-4a6d-a2df-b40c54459249" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kserve;K8S_POD_NAME=odh-model-controller-84b54b7f97-mhmm6;K8S_POD_INFRA_CONTAINER_ID=fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf;K8S_POD_UID=aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd" Path:"" ERRORED: error configuring pod [kserve/odh-model-controller-84b54b7f97-mhmm6] networking: Multus: [kserve/odh-model-controller-84b54b7f97-mhmm6/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd]: error waiting for pod: Get "https://a02a453c93518445492c708b3b388bf1-2230622eb921ce79.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kserve/pods/odh-model-controller-84b54b7f97-mhmm6?timeout=1m0s": context deadline exceeded
Jul 06 20:45:28.833822 ip-10-0-134-39 kubenswrapper[2541]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 06 20:45:28.833822 ip-10-0-134-39 kubenswrapper[2541]:  >
Jul 06 20:45:28.834197 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:45:28.833844    2541 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 06 20:45:28.834197 ip-10-0-134-39 kubenswrapper[2541]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-84b54b7f97-mhmm6_kserve_aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd_0(fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf): error adding pod kserve_odh-model-controller-84b54b7f97-mhmm6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf" Netns:"/var/run/netns/8101b75d-944e-4a6d-a2df-b40c54459249" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kserve;K8S_POD_NAME=odh-model-controller-84b54b7f97-mhmm6;K8S_POD_INFRA_CONTAINER_ID=fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf;K8S_POD_UID=aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd" Path:"" ERRORED: error configuring pod [kserve/odh-model-controller-84b54b7f97-mhmm6] networking: Multus: [kserve/odh-model-controller-84b54b7f97-mhmm6/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd]: error waiting for pod: Get "https://a02a453c93518445492c708b3b388bf1-2230622eb921ce79.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kserve/pods/odh-model-controller-84b54b7f97-mhmm6?timeout=1m0s": context deadline exceeded
Jul 06 20:45:28.834197 ip-10-0-134-39 kubenswrapper[2541]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 06 20:45:28.834197 ip-10-0-134-39 kubenswrapper[2541]:  > pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:28.834197 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:45:28.833869    2541 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 06 20:45:28.834197 ip-10-0-134-39 kubenswrapper[2541]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-84b54b7f97-mhmm6_kserve_aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd_0(fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf): error adding pod kserve_odh-model-controller-84b54b7f97-mhmm6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf" Netns:"/var/run/netns/8101b75d-944e-4a6d-a2df-b40c54459249" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kserve;K8S_POD_NAME=odh-model-controller-84b54b7f97-mhmm6;K8S_POD_INFRA_CONTAINER_ID=fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf;K8S_POD_UID=aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd" Path:"" ERRORED: error configuring pod [kserve/odh-model-controller-84b54b7f97-mhmm6] networking: Multus: [kserve/odh-model-controller-84b54b7f97-mhmm6/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd]: error waiting for pod: Get "https://a02a453c93518445492c708b3b388bf1-2230622eb921ce79.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kserve/pods/odh-model-controller-84b54b7f97-mhmm6?timeout=1m0s": context deadline exceeded
Jul 06 20:45:28.834197 ip-10-0-134-39 kubenswrapper[2541]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 06 20:45:28.834197 ip-10-0-134-39 kubenswrapper[2541]:  > pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:28.834197 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:45:28.833944    2541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"odh-model-controller-84b54b7f97-mhmm6_kserve(aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"odh-model-controller-84b54b7f97-mhmm6_kserve(aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-84b54b7f97-mhmm6_kserve_aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd_0(fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf): error adding pod kserve_odh-model-controller-84b54b7f97-mhmm6 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf\\\" Netns:\\\"/var/run/netns/8101b75d-944e-4a6d-a2df-b40c54459249\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=kserve;K8S_POD_NAME=odh-model-controller-84b54b7f97-mhmm6;K8S_POD_INFRA_CONTAINER_ID=fe0587f360a2d137780173f734ee474e4a2e617191a1e639adaa915a846f0cbf;K8S_POD_UID=aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd\\\" Path:\\\"\\\" ERRORED: error configuring pod [kserve/odh-model-controller-84b54b7f97-mhmm6] networking: Multus: [kserve/odh-model-controller-84b54b7f97-mhmm6/aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd]: error waiting for pod: Get \\\"https://a02a453c93518445492c708b3b388bf1-2230622eb921ce79.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kserve/pods/odh-model-controller-84b54b7f97-mhmm6?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="kserve/odh-model-controller-84b54b7f97-mhmm6" podUID="aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd"
Jul 06 20:45:42.087938 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:42.087912    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:42.088370 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:42.088115    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:42.207617 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:42.207592    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-mhmm6"]
Jul 06 20:45:42.210243 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:45:42.210219    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1bf996_1c87_4fac_adf7_a42b2c6ed4dd.slice/crio-525726fd30f15e89253565e5fa517790de3b21d542716f2bccce6c09916ed113 WatchSource:0}: Error finding container 525726fd30f15e89253565e5fa517790de3b21d542716f2bccce6c09916ed113: Status 404 returned error can't find the container with id 525726fd30f15e89253565e5fa517790de3b21d542716f2bccce6c09916ed113
Jul 06 20:45:42.302076 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:42.302030    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-mhmm6" event={"ID":"aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd","Type":"ContainerStarted","Data":"525726fd30f15e89253565e5fa517790de3b21d542716f2bccce6c09916ed113"}
Jul 06 20:45:45.317258 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:45.317218    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-mhmm6" event={"ID":"aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd","Type":"ContainerStarted","Data":"6d8eb9426229e2318cd2df439d436b330fa03f577cc75cfea5fc290b48987e09"}
Jul 06 20:45:45.317669 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:45.317334    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:45.332457 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:45.332412    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-mhmm6" podStartSLOduration=19.655883106 podStartE2EDuration="22.332399738s" totalImagesPullingTime="2.676516632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:45:45.331684653 +0000 UTC m=+620.673587369" watchObservedRunningTime="2026-07-06 20:45:45.332399738 +0000 UTC m=+620.674302492"
Jul 06 20:45:56.323131 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:56.323031    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-mhmm6"
Jul 06 20:45:57.160667 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.160641    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-t4ndd"]
Jul 06 20:45:57.165271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.165242    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t4ndd"
Jul 06 20:45:57.170575 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.170551    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-t4ndd"]
Jul 06 20:45:57.233351 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.233330    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cj4t\" (UniqueName: \"kubernetes.io/projected/10e8ceda-967f-43e4-9cf1-b78a6f78e683-kube-api-access-6cj4t\") pod \"s3-init-t4ndd\" (UID: \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\") " pod="kserve/s3-init-t4ndd"
Jul 06 20:45:57.233461 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.233364    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e8ceda-967f-43e4-9cf1-b78a6f78e683-model-cache\") pod \"s3-init-t4ndd\" (UID: \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\") " pod="kserve/s3-init-t4ndd"
Jul 06 20:45:57.333646 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.333623    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cj4t\" (UniqueName: \"kubernetes.io/projected/10e8ceda-967f-43e4-9cf1-b78a6f78e683-kube-api-access-6cj4t\") pod \"s3-init-t4ndd\" (UID: \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\") " pod="kserve/s3-init-t4ndd"
Jul 06 20:45:57.333972 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.333657    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e8ceda-967f-43e4-9cf1-b78a6f78e683-model-cache\") pod \"s3-init-t4ndd\" (UID: \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\") " pod="kserve/s3-init-t4ndd"
Jul 06 20:45:57.333972 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.333926    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e8ceda-967f-43e4-9cf1-b78a6f78e683-model-cache\") pod \"s3-init-t4ndd\" (UID: \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\") " pod="kserve/s3-init-t4ndd"
Jul 06 20:45:57.341383 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.341359    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cj4t\" (UniqueName: \"kubernetes.io/projected/10e8ceda-967f-43e4-9cf1-b78a6f78e683-kube-api-access-6cj4t\") pod \"s3-init-t4ndd\" (UID: \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\") " pod="kserve/s3-init-t4ndd"
Jul 06 20:45:57.486640 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.486620    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t4ndd"
Jul 06 20:45:57.611829 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:57.611680    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-t4ndd"]
Jul 06 20:45:57.614262 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:45:57.614231    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10e8ceda_967f_43e4_9cf1_b78a6f78e683.slice/crio-fd04226e78f219dfe7432c7a1c13c635ac38f8fb6e4813bd79cacb79e4654134 WatchSource:0}: Error finding container fd04226e78f219dfe7432c7a1c13c635ac38f8fb6e4813bd79cacb79e4654134: Status 404 returned error can't find the container with id fd04226e78f219dfe7432c7a1c13c635ac38f8fb6e4813bd79cacb79e4654134
Jul 06 20:45:58.373983 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:45:58.373942    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t4ndd" event={"ID":"10e8ceda-967f-43e4-9cf1-b78a6f78e683","Type":"ContainerStarted","Data":"fd04226e78f219dfe7432c7a1c13c635ac38f8fb6e4813bd79cacb79e4654134"}
Jul 06 20:46:01.386834 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:46:01.386762    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t4ndd" event={"ID":"10e8ceda-967f-43e4-9cf1-b78a6f78e683","Type":"ContainerStarted","Data":"35400ae55331f980f443c0a761ba72753b7f7581546707960f87f4f33f1657cd"}
Jul 06 20:46:03.395887 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:46:03.395857    2541 generic.go:356] "Generic (PLEG): container finished" podID="10e8ceda-967f-43e4-9cf1-b78a6f78e683" containerID="35400ae55331f980f443c0a761ba72753b7f7581546707960f87f4f33f1657cd" exitCode=0
Jul 06 20:46:03.396221 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:46:03.395898    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t4ndd" event={"ID":"10e8ceda-967f-43e4-9cf1-b78a6f78e683","Type":"ContainerDied","Data":"35400ae55331f980f443c0a761ba72753b7f7581546707960f87f4f33f1657cd"}
Jul 06 20:46:08.418767 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:46:08.418673    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t4ndd" event={"ID":"10e8ceda-967f-43e4-9cf1-b78a6f78e683","Type":"ContainerStarted","Data":"5cb60c601b6efe30587c030f9604b1c5e93942517c4dc07960025931b1085dff"}
Jul 06 20:46:08.437401 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:46:08.437347    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-t4ndd" podStartSLOduration=1.843727221 podStartE2EDuration="11.437332019s" totalImagesPullingTime="8.055875242s" totalInitContainerRuntime="1.537729556s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:45:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:08.436124108 +0000 UTC m=+643.778026824" watchObservedRunningTime="2026-07-06 20:46:08.437332019 +0000 UTC m=+643.779234733"
Jul 06 20:47:12.653316 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:12.653283    2541 generic.go:356] "Generic (PLEG): container finished" podID="10e8ceda-967f-43e4-9cf1-b78a6f78e683" containerID="5cb60c601b6efe30587c030f9604b1c5e93942517c4dc07960025931b1085dff" exitCode=0
Jul 06 20:47:12.653711 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:12.653327    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t4ndd" event={"ID":"10e8ceda-967f-43e4-9cf1-b78a6f78e683","Type":"ContainerDied","Data":"5cb60c601b6efe30587c030f9604b1c5e93942517c4dc07960025931b1085dff"}
Jul 06 20:47:13.773975 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:13.773951    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t4ndd"
Jul 06 20:47:13.861113 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:13.861091    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e8ceda-967f-43e4-9cf1-b78a6f78e683-model-cache\") pod \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\" (UID: \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\") "
Jul 06 20:47:13.861208 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:13.861131    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cj4t\" (UniqueName: \"kubernetes.io/projected/10e8ceda-967f-43e4-9cf1-b78a6f78e683-kube-api-access-6cj4t\") pod \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\" (UID: \"10e8ceda-967f-43e4-9cf1-b78a6f78e683\") "
Jul 06 20:47:13.863280 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:13.863248    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e8ceda-967f-43e4-9cf1-b78a6f78e683-kube-api-access-6cj4t" (OuterVolumeSpecName: "kube-api-access-6cj4t") pod "10e8ceda-967f-43e4-9cf1-b78a6f78e683" (UID: "10e8ceda-967f-43e4-9cf1-b78a6f78e683"). InnerVolumeSpecName "kube-api-access-6cj4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:47:13.879023 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:13.879002    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e8ceda-967f-43e4-9cf1-b78a6f78e683-model-cache" (OuterVolumeSpecName: "model-cache") pod "10e8ceda-967f-43e4-9cf1-b78a6f78e683" (UID: "10e8ceda-967f-43e4-9cf1-b78a6f78e683"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:47:13.961964 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:13.961919    2541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10e8ceda-967f-43e4-9cf1-b78a6f78e683-model-cache\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:47:13.961964 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:13.961938    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cj4t\" (UniqueName: \"kubernetes.io/projected/10e8ceda-967f-43e4-9cf1-b78a6f78e683-kube-api-access-6cj4t\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:47:14.661420 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:14.661397    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-t4ndd" event={"ID":"10e8ceda-967f-43e4-9cf1-b78a6f78e683","Type":"ContainerDied","Data":"fd04226e78f219dfe7432c7a1c13c635ac38f8fb6e4813bd79cacb79e4654134"}
Jul 06 20:47:14.661507 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:14.661426    2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd04226e78f219dfe7432c7a1c13c635ac38f8fb6e4813bd79cacb79e4654134"
Jul 06 20:47:14.661507 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:14.661407    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-t4ndd"
Jul 06 20:47:57.138565 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.138534    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-6tg6v"]
Jul 06 20:47:57.139014 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.138820    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e8ceda-967f-43e4-9cf1-b78a6f78e683" containerName="download-hf-model"
Jul 06 20:47:57.139014 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.138830    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e8ceda-967f-43e4-9cf1-b78a6f78e683" containerName="download-hf-model"
Jul 06 20:47:57.139014 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.138839    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e8ceda-967f-43e4-9cf1-b78a6f78e683" containerName="s3-init"
Jul 06 20:47:57.139014 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.138844    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e8ceda-967f-43e4-9cf1-b78a6f78e683" containerName="s3-init"
Jul 06 20:47:57.139014 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.138891    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e8ceda-967f-43e4-9cf1-b78a6f78e683" containerName="s3-init"
Jul 06 20:47:57.141780 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.141766    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:47:57.143787 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.143766    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 06 20:47:57.149253 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.149231    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6tg6v"]
Jul 06 20:47:57.152155 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.152135    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d29ed2-92fe-4192-96b6-20b4326cc754-model-cache\") pod \"s3-tls-init-custom-6tg6v\" (UID: \"88d29ed2-92fe-4192-96b6-20b4326cc754\") " pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:47:57.152232 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.152184    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjcj\" (UniqueName: \"kubernetes.io/projected/88d29ed2-92fe-4192-96b6-20b4326cc754-kube-api-access-2qjcj\") pod \"s3-tls-init-custom-6tg6v\" (UID: \"88d29ed2-92fe-4192-96b6-20b4326cc754\") " pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:47:57.252458 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.252438    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d29ed2-92fe-4192-96b6-20b4326cc754-model-cache\") pod \"s3-tls-init-custom-6tg6v\" (UID: \"88d29ed2-92fe-4192-96b6-20b4326cc754\") " pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:47:57.252547 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.252474    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjcj\" (UniqueName: \"kubernetes.io/projected/88d29ed2-92fe-4192-96b6-20b4326cc754-kube-api-access-2qjcj\") pod \"s3-tls-init-custom-6tg6v\" (UID: \"88d29ed2-92fe-4192-96b6-20b4326cc754\") " pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:47:57.252754 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.252738    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d29ed2-92fe-4192-96b6-20b4326cc754-model-cache\") pod \"s3-tls-init-custom-6tg6v\" (UID: \"88d29ed2-92fe-4192-96b6-20b4326cc754\") " pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:47:57.259418 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.259394    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjcj\" (UniqueName: \"kubernetes.io/projected/88d29ed2-92fe-4192-96b6-20b4326cc754-kube-api-access-2qjcj\") pod \"s3-tls-init-custom-6tg6v\" (UID: \"88d29ed2-92fe-4192-96b6-20b4326cc754\") " pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:47:57.463183 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.463109    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:47:57.584025 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.584001    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-6tg6v"]
Jul 06 20:47:57.586939 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:47:57.586899    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88d29ed2_92fe_4192_96b6_20b4326cc754.slice/crio-ca3adb7c3aee84cdd6ff873867946d5e1ddc14be8b94e3c3feb19bb6fb36ae14 WatchSource:0}: Error finding container ca3adb7c3aee84cdd6ff873867946d5e1ddc14be8b94e3c3feb19bb6fb36ae14: Status 404 returned error can't find the container with id ca3adb7c3aee84cdd6ff873867946d5e1ddc14be8b94e3c3feb19bb6fb36ae14
Jul 06 20:47:57.818956 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.818923    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6tg6v" event={"ID":"88d29ed2-92fe-4192-96b6-20b4326cc754","Type":"ContainerStarted","Data":"97d9ed76652d6c9cd6709dd9599dfc11450631bfdd4cfb4b3513ba22bbfe59c2"}
Jul 06 20:47:57.819134 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:57.818971    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6tg6v" event={"ID":"88d29ed2-92fe-4192-96b6-20b4326cc754","Type":"ContainerStarted","Data":"ca3adb7c3aee84cdd6ff873867946d5e1ddc14be8b94e3c3feb19bb6fb36ae14"}
Jul 06 20:47:59.828203 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:59.828175    2541 generic.go:356] "Generic (PLEG): container finished" podID="88d29ed2-92fe-4192-96b6-20b4326cc754" containerID="97d9ed76652d6c9cd6709dd9599dfc11450631bfdd4cfb4b3513ba22bbfe59c2" exitCode=0
Jul 06 20:47:59.828481 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:47:59.828249    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6tg6v" event={"ID":"88d29ed2-92fe-4192-96b6-20b4326cc754","Type":"ContainerDied","Data":"97d9ed76652d6c9cd6709dd9599dfc11450631bfdd4cfb4b3513ba22bbfe59c2"}
Jul 06 20:48:00.838564 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:00.838521    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6tg6v" event={"ID":"88d29ed2-92fe-4192-96b6-20b4326cc754","Type":"ContainerStarted","Data":"dbc29d043d0fe0b21e957f176a3bc6ad68dbb28acc8212da64c9540a9bcad45b"}
Jul 06 20:48:00.854827 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:00.854787    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-6tg6v" podStartSLOduration=2.545251194 podStartE2EDuration="3.854771438s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.309520244s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:00.853327289 +0000 UTC m=+756.195230006" watchObservedRunningTime="2026-07-06 20:48:00.854771438 +0000 UTC m=+756.196674154"
Jul 06 20:48:05.867888 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:05.867861    2541 generic.go:356] "Generic (PLEG): container finished" podID="88d29ed2-92fe-4192-96b6-20b4326cc754" containerID="dbc29d043d0fe0b21e957f176a3bc6ad68dbb28acc8212da64c9540a9bcad45b" exitCode=0
Jul 06 20:48:05.868191 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:05.867936    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6tg6v" event={"ID":"88d29ed2-92fe-4192-96b6-20b4326cc754","Type":"ContainerDied","Data":"dbc29d043d0fe0b21e957f176a3bc6ad68dbb28acc8212da64c9540a9bcad45b"}
Jul 06 20:48:07.012530 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.012506    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:48:07.024904 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.024874    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d29ed2-92fe-4192-96b6-20b4326cc754-model-cache\") pod \"88d29ed2-92fe-4192-96b6-20b4326cc754\" (UID: \"88d29ed2-92fe-4192-96b6-20b4326cc754\") "
Jul 06 20:48:07.025069 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.024910    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qjcj\" (UniqueName: \"kubernetes.io/projected/88d29ed2-92fe-4192-96b6-20b4326cc754-kube-api-access-2qjcj\") pod \"88d29ed2-92fe-4192-96b6-20b4326cc754\" (UID: \"88d29ed2-92fe-4192-96b6-20b4326cc754\") "
Jul 06 20:48:07.027062 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.026996    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d29ed2-92fe-4192-96b6-20b4326cc754-kube-api-access-2qjcj" (OuterVolumeSpecName: "kube-api-access-2qjcj") pod "88d29ed2-92fe-4192-96b6-20b4326cc754" (UID: "88d29ed2-92fe-4192-96b6-20b4326cc754"). InnerVolumeSpecName "kube-api-access-2qjcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:48:07.048835 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.048810    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d29ed2-92fe-4192-96b6-20b4326cc754-model-cache" (OuterVolumeSpecName: "model-cache") pod "88d29ed2-92fe-4192-96b6-20b4326cc754" (UID: "88d29ed2-92fe-4192-96b6-20b4326cc754"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:07.126087 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.126017    2541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d29ed2-92fe-4192-96b6-20b4326cc754-model-cache\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:48:07.126087 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.126038    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qjcj\" (UniqueName: \"kubernetes.io/projected/88d29ed2-92fe-4192-96b6-20b4326cc754-kube-api-access-2qjcj\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:48:07.876207 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.876181    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-6tg6v" event={"ID":"88d29ed2-92fe-4192-96b6-20b4326cc754","Type":"ContainerDied","Data":"ca3adb7c3aee84cdd6ff873867946d5e1ddc14be8b94e3c3feb19bb6fb36ae14"}
Jul 06 20:48:07.876301 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.876214    2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3adb7c3aee84cdd6ff873867946d5e1ddc14be8b94e3c3feb19bb6fb36ae14"
Jul 06 20:48:07.876301 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:07.876218    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-6tg6v"
Jul 06 20:48:10.582347 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.582318    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-6d4kf"]
Jul 06 20:48:10.582689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.582600    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d29ed2-92fe-4192-96b6-20b4326cc754" containerName="s3-tls-init-custom"
Jul 06 20:48:10.582689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.582610    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d29ed2-92fe-4192-96b6-20b4326cc754" containerName="s3-tls-init-custom"
Jul 06 20:48:10.582689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.582627    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d29ed2-92fe-4192-96b6-20b4326cc754" containerName="download-hf-model"
Jul 06 20:48:10.582689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.582634    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d29ed2-92fe-4192-96b6-20b4326cc754" containerName="download-hf-model"
Jul 06 20:48:10.582689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.582680    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d29ed2-92fe-4192-96b6-20b4326cc754" containerName="s3-tls-init-custom"
Jul 06 20:48:10.586612 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.586596    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:10.588371 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.588351    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 06 20:48:10.592470 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.592453    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-6d4kf"]
Jul 06 20:48:10.650615 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.650597    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrl6z\" (UniqueName: \"kubernetes.io/projected/55b448e7-eae9-4325-8431-6133280eb6e9-kube-api-access-wrl6z\") pod \"s3-tls-init-serving-6d4kf\" (UID: \"55b448e7-eae9-4325-8431-6133280eb6e9\") " pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:10.650703 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.650642    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b448e7-eae9-4325-8431-6133280eb6e9-model-cache\") pod \"s3-tls-init-serving-6d4kf\" (UID: \"55b448e7-eae9-4325-8431-6133280eb6e9\") " pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:10.750876 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.750852    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b448e7-eae9-4325-8431-6133280eb6e9-model-cache\") pod \"s3-tls-init-serving-6d4kf\" (UID: \"55b448e7-eae9-4325-8431-6133280eb6e9\") " pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:10.750959 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.750894    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrl6z\" (UniqueName: \"kubernetes.io/projected/55b448e7-eae9-4325-8431-6133280eb6e9-kube-api-access-wrl6z\") pod \"s3-tls-init-serving-6d4kf\" (UID: \"55b448e7-eae9-4325-8431-6133280eb6e9\") " pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:10.751203 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.751186    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b448e7-eae9-4325-8431-6133280eb6e9-model-cache\") pod \"s3-tls-init-serving-6d4kf\" (UID: \"55b448e7-eae9-4325-8431-6133280eb6e9\") " pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:10.757788 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.757765    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrl6z\" (UniqueName: \"kubernetes.io/projected/55b448e7-eae9-4325-8431-6133280eb6e9-kube-api-access-wrl6z\") pod \"s3-tls-init-serving-6d4kf\" (UID: \"55b448e7-eae9-4325-8431-6133280eb6e9\") " pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:10.904119 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:10.904076    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:11.025688 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:11.025634    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-6d4kf"]
Jul 06 20:48:11.027725 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:48:11.027692    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55b448e7_eae9_4325_8431_6133280eb6e9.slice/crio-d58d46996755c8d6f0eef454bb4beeeab9e436bf954d9e6e4109717538b37db4 WatchSource:0}: Error finding container d58d46996755c8d6f0eef454bb4beeeab9e436bf954d9e6e4109717538b37db4: Status 404 returned error can't find the container with id d58d46996755c8d6f0eef454bb4beeeab9e436bf954d9e6e4109717538b37db4
Jul 06 20:48:11.893313 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:11.893273    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6d4kf" event={"ID":"55b448e7-eae9-4325-8431-6133280eb6e9","Type":"ContainerStarted","Data":"cbeab42ed10618122964fef6378d85c200385046ace7db403801bf29f2218f76"}
Jul 06 20:48:11.893313 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:11.893318    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6d4kf" event={"ID":"55b448e7-eae9-4325-8431-6133280eb6e9","Type":"ContainerStarted","Data":"d58d46996755c8d6f0eef454bb4beeeab9e436bf954d9e6e4109717538b37db4"}
Jul 06 20:48:13.901958 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:13.901931    2541 generic.go:356] "Generic (PLEG): container finished" podID="55b448e7-eae9-4325-8431-6133280eb6e9" containerID="cbeab42ed10618122964fef6378d85c200385046ace7db403801bf29f2218f76" exitCode=0
Jul 06 20:48:13.902290 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:13.902005    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6d4kf" event={"ID":"55b448e7-eae9-4325-8431-6133280eb6e9","Type":"ContainerDied","Data":"cbeab42ed10618122964fef6378d85c200385046ace7db403801bf29f2218f76"}
Jul 06 20:48:14.907821 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:14.907788    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6d4kf" event={"ID":"55b448e7-eae9-4325-8431-6133280eb6e9","Type":"ContainerStarted","Data":"bc603f6933d9bf53993425456d3c0e220d5badf0cb60e81800f4ab491e2db539"}
Jul 06 20:48:14.924723 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:14.924683    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-6d4kf" podStartSLOduration=3.027590385 podStartE2EDuration="4.924670289s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.897079904s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:14.92261974 +0000 UTC m=+770.264522457" watchObservedRunningTime="2026-07-06 20:48:14.924670289 +0000 UTC m=+770.266573004"
Jul 06 20:48:19.928892 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:19.928861    2541 generic.go:356] "Generic (PLEG): container finished" podID="55b448e7-eae9-4325-8431-6133280eb6e9" containerID="bc603f6933d9bf53993425456d3c0e220d5badf0cb60e81800f4ab491e2db539" exitCode=0
Jul 06 20:48:19.929175 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:19.928931    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6d4kf" event={"ID":"55b448e7-eae9-4325-8431-6133280eb6e9","Type":"ContainerDied","Data":"bc603f6933d9bf53993425456d3c0e220d5badf0cb60e81800f4ab491e2db539"}
Jul 06 20:48:21.081225 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.081203    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:21.138149 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.138117    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b448e7-eae9-4325-8431-6133280eb6e9-model-cache\") pod \"55b448e7-eae9-4325-8431-6133280eb6e9\" (UID: \"55b448e7-eae9-4325-8431-6133280eb6e9\") "
Jul 06 20:48:21.138290 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.138206    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrl6z\" (UniqueName: \"kubernetes.io/projected/55b448e7-eae9-4325-8431-6133280eb6e9-kube-api-access-wrl6z\") pod \"55b448e7-eae9-4325-8431-6133280eb6e9\" (UID: \"55b448e7-eae9-4325-8431-6133280eb6e9\") "
Jul 06 20:48:21.140247 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.140223    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b448e7-eae9-4325-8431-6133280eb6e9-kube-api-access-wrl6z" (OuterVolumeSpecName: "kube-api-access-wrl6z") pod "55b448e7-eae9-4325-8431-6133280eb6e9" (UID: "55b448e7-eae9-4325-8431-6133280eb6e9"). InnerVolumeSpecName "kube-api-access-wrl6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:48:21.160866 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.160839    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b448e7-eae9-4325-8431-6133280eb6e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "55b448e7-eae9-4325-8431-6133280eb6e9" (UID: "55b448e7-eae9-4325-8431-6133280eb6e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:48:21.239004 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.238984    2541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b448e7-eae9-4325-8431-6133280eb6e9-model-cache\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:48:21.239004 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.239005    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrl6z\" (UniqueName: \"kubernetes.io/projected/55b448e7-eae9-4325-8431-6133280eb6e9-kube-api-access-wrl6z\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:48:21.938020 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.937999    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6d4kf"
Jul 06 20:48:21.938020 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.938011    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6d4kf" event={"ID":"55b448e7-eae9-4325-8431-6133280eb6e9","Type":"ContainerDied","Data":"d58d46996755c8d6f0eef454bb4beeeab9e436bf954d9e6e4109717538b37db4"}
Jul 06 20:48:21.938222 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:21.938039    2541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d58d46996755c8d6f0eef454bb4beeeab9e436bf954d9e6e4109717538b37db4"
Jul 06 20:48:33.672024 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.671994    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"]
Jul 06 20:48:33.672494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.672287    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b448e7-eae9-4325-8431-6133280eb6e9" containerName="download-hf-model"
Jul 06 20:48:33.672494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.672297    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b448e7-eae9-4325-8431-6133280eb6e9" containerName="download-hf-model"
Jul 06 20:48:33.672494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.672320    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b448e7-eae9-4325-8431-6133280eb6e9" containerName="s3-tls-init-serving"
Jul 06 20:48:33.672494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.672326    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b448e7-eae9-4325-8431-6133280eb6e9" containerName="s3-tls-init-serving"
Jul 06 20:48:33.672494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.672371    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b448e7-eae9-4325-8431-6133280eb6e9" containerName="s3-tls-init-serving"
Jul 06 20:48:33.677644 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.677628    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.679500 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.679472    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 20:48:33.679638 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.679502    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 06 20:48:33.679638 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.679472    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:48:33.679638 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.679611    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 20:48:33.679818 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.679800    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 06 20:48:33.680128 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.680114    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 20:48:33.680206 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.680143    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5qwfj\""
Jul 06 20:48:33.684769 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.684751    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"]
Jul 06 20:48:33.732234 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.732210    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.732324 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.732238    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.732324 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.732260    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c14dd981-8a33-4993-88c5-dc90d42c378c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.732406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.732349    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dhz\" (UniqueName: \"kubernetes.io/projected/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-api-access-j7dhz\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.732406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.732391    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c14dd981-8a33-4993-88c5-dc90d42c378c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.833490 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.833472    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dhz\" (UniqueName: \"kubernetes.io/projected/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-api-access-j7dhz\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.833590 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.833499    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c14dd981-8a33-4993-88c5-dc90d42c378c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.833590 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.833520    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.833590 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.833539    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.833590 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.833560    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c14dd981-8a33-4993-88c5-dc90d42c378c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.833967 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.833949    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c14dd981-8a33-4993-88c5-dc90d42c378c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.834242 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.834219    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.834322 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.834282    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.835997 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.835982    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c14dd981-8a33-4993-88c5-dc90d42c378c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.840369 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.840350    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dhz\" (UniqueName: \"kubernetes.io/projected/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-api-access-j7dhz\") pod \"isvc-sklearn-batcher-predictor-8659b9c9df-jw679\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:33.988456 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:33.988435    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:34.114840 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:34.114812    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"]
Jul 06 20:48:34.118567 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:48:34.118541    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14dd981_8a33_4993_88c5_dc90d42c378c.slice/crio-94621c4f5d8d5ec4d7d9fbe9efab2f38b662489466bd3b6f569d149dc1ccdd99 WatchSource:0}: Error finding container 94621c4f5d8d5ec4d7d9fbe9efab2f38b662489466bd3b6f569d149dc1ccdd99: Status 404 returned error can't find the container with id 94621c4f5d8d5ec4d7d9fbe9efab2f38b662489466bd3b6f569d149dc1ccdd99
Jul 06 20:48:34.986570 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:34.986535    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerStarted","Data":"ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2"}
Jul 06 20:48:34.986901 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:34.986623    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerStarted","Data":"94621c4f5d8d5ec4d7d9fbe9efab2f38b662489466bd3b6f569d149dc1ccdd99"}
Jul 06 20:48:38.001343 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:38.001313    2541 generic.go:356] "Generic (PLEG): container finished" podID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerID="ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2" exitCode=0
Jul 06 20:48:38.001716 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:38.001394    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerDied","Data":"ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2"}
Jul 06 20:48:53.095999 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:53.095957    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerStarted","Data":"eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc"}
Jul 06 20:48:55.108239 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:55.108200    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerStarted","Data":"8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88"}
Jul 06 20:48:58.124366 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:58.124296    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerStarted","Data":"2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3"}
Jul 06 20:48:58.124779 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:58.124507    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:58.124779 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:58.124678    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:58.125649 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:58.125621    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:48:58.144648 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:58.144603    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podStartSLOduration=1.876596346 podStartE2EDuration="25.144591859s" totalImagesPullingTime="19.666487439s" totalInitContainerRuntime="3.601508074s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:58.143473118 +0000 UTC m=+813.485375833" watchObservedRunningTime="2026-07-06 20:48:58.144591859 +0000 UTC m=+813.486494574"
Jul 06 20:48:59.128135 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:59.128111    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:48:59.128511 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:59.128187    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:48:59.129020 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:59.128998    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:48:59.131951 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:48:59.131936    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:49:00.132353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:00.132322    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:49:00.132707 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:00.132647    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:49:01.135911 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:01.135864    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:49:01.136313 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:01.136170    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:49:11.136707 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:11.136672    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:49:11.137230 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:11.137160    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:49:21.136784 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:21.136735    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:49:21.137233 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:21.137148    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:49:31.135877 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:31.135833    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:49:31.136332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:31.136277    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:49:41.136238 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:41.136195    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:49:41.136643 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:41.136621    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:49:51.136081 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:51.136021    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:49:51.136546 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:49:51.136424    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:01.136782 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:01.136754    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:50:01.137143 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:01.136974    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:50:08.867294 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:08.867265    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"]
Jul 06 20:50:08.867701 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:08.867480    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" containerID="cri-o://eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc" gracePeriod=30
Jul 06 20:50:08.867701 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:08.867522    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" containerID="cri-o://2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3" gracePeriod=30
Jul 06 20:50:08.867701 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:08.867547    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy" containerID="cri-o://8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88" gracePeriod=30
Jul 06 20:50:08.891427 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:08.891404    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"]
Jul 06 20:50:08.902630 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:08.902606    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:08.904807 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:08.904667    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 06 20:50:08.904954 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:08.904933    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 06 20:50:08.906522 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:08.906504    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"]
Jul 06 20:50:09.016458 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.016428    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9b7dcd-130e-4a50-bf1e-cd184790398d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.016599 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.016481    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9tw\" (UniqueName: \"kubernetes.io/projected/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-api-access-ld9tw\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.016599 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.016557    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.016719 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.016620    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.016719 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.016668    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.117284 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.117254    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.117284 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.117286    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9b7dcd-130e-4a50-bf1e-cd184790398d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.117503 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.117310    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9tw\" (UniqueName: \"kubernetes.io/projected/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-api-access-ld9tw\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.117503 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.117329    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.117503 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.117356    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.117503 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:50:09.117408    2541 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 06 20:50:09.117503 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:50:09.117485    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a9b7dcd-130e-4a50-bf1e-cd184790398d-proxy-tls podName:4a9b7dcd-130e-4a50-bf1e-cd184790398d nodeName:}" failed. No retries permitted until 2026-07-06 20:50:09.617465907 +0000 UTC m=+884.959368604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a9b7dcd-130e-4a50-bf1e-cd184790398d-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" (UID: "4a9b7dcd-130e-4a50-bf1e-cd184790398d") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 06 20:50:09.117757 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.117649    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.117937 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.117918    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.117937 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.117929    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.124894 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.124868    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9tw\" (UniqueName: \"kubernetes.io/projected/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-api-access-ld9tw\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.128946 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.128920    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 06 20:50:09.425480 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.425428    2541 generic.go:356] "Generic (PLEG): container finished" podID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerID="8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88" exitCode=2
Jul 06 20:50:09.425568 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.425497    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerDied","Data":"8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88"}
Jul 06 20:50:09.621226 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.621198    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9b7dcd-130e-4a50-bf1e-cd184790398d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.623568 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.623551    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9b7dcd-130e-4a50-bf1e-cd184790398d-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.815732 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.815707    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:09.940841 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:09.940616    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"]
Jul 06 20:50:09.943457 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:50:09.943427    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9b7dcd_130e_4a50_bf1e_cd184790398d.slice/crio-dac37db2817f3fc014f8e752673d7f4cffff086f8573cb5cde432fe5c8d45e7c WatchSource:0}: Error finding container dac37db2817f3fc014f8e752673d7f4cffff086f8573cb5cde432fe5c8d45e7c: Status 404 returned error can't find the container with id dac37db2817f3fc014f8e752673d7f4cffff086f8573cb5cde432fe5c8d45e7c
Jul 06 20:50:10.430958 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:10.430921    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerStarted","Data":"cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd"}
Jul 06 20:50:10.431133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:10.431000    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerStarted","Data":"dac37db2817f3fc014f8e752673d7f4cffff086f8573cb5cde432fe5c8d45e7c"}
Jul 06 20:50:11.135856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:11.135813    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:50:11.136271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:11.136146    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:13.447206 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:13.447170    2541 generic.go:356] "Generic (PLEG): container finished" podID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerID="eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc" exitCode=0
Jul 06 20:50:13.447518 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:13.447238    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerDied","Data":"eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc"}
Jul 06 20:50:14.128869 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:14.128812    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 06 20:50:14.452770 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:14.452712    2541 generic.go:356] "Generic (PLEG): container finished" podID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerID="cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd" exitCode=0
Jul 06 20:50:14.453037 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:14.452789    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerDied","Data":"cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd"}
Jul 06 20:50:15.459532 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:15.459500    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerStarted","Data":"cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf"}
Jul 06 20:50:15.459916 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:15.459562    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerStarted","Data":"0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee"}
Jul 06 20:50:15.459916 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:15.459587    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerStarted","Data":"85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e"}
Jul 06 20:50:15.459916 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:15.459848    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:15.460036 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:15.459994    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:15.461086 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:15.461037    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:50:15.488201 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:15.487621    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podStartSLOduration=3.70358302 podStartE2EDuration="7.487608294s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.784025274s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:50:15.486429536 +0000 UTC m=+890.828332252" watchObservedRunningTime="2026-07-06 20:50:15.487608294 +0000 UTC m=+890.829511009"
Jul 06 20:50:16.464354 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:16.464260    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:50:16.464354 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:16.464297    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:16.465257 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:16.465226    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:17.469019 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:17.468978    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:50:17.469450 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:17.469347    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:19.129214 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:19.129177    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 06 20:50:19.129598 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:19.129267    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:50:21.136087 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:21.136021    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:50:21.136482 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:21.136371    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:22.472924 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:22.472896    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:50:22.473515 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:22.473487    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:50:22.473650 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:22.473625    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:24.128911 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:24.128872    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 06 20:50:29.129171 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:29.129131    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 06 20:50:31.136303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:31.136264    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 06 20:50:31.136692 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:31.136399    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:50:31.136692 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:31.136553    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:31.136692 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:31.136633    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:50:32.473397 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:32.473351    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:50:32.473822 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:32.473780    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:34.128348 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:34.128310    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 06 20:50:39.004600 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.004578    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:50:39.129119 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.129037    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7dhz\" (UniqueName: \"kubernetes.io/projected/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-api-access-j7dhz\") pod \"c14dd981-8a33-4993-88c5-dc90d42c378c\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") "
Jul 06 20:50:39.129119 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.129095    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-rbac-proxy-sar-config\") pod \"c14dd981-8a33-4993-88c5-dc90d42c378c\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") "
Jul 06 20:50:39.129303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.129122    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c14dd981-8a33-4993-88c5-dc90d42c378c-proxy-tls\") pod \"c14dd981-8a33-4993-88c5-dc90d42c378c\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") "
Jul 06 20:50:39.129303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.129163    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c14dd981-8a33-4993-88c5-dc90d42c378c-kserve-provision-location\") pod \"c14dd981-8a33-4993-88c5-dc90d42c378c\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") "
Jul 06 20:50:39.129303 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.129186    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-cabundle-cert\") pod \"c14dd981-8a33-4993-88c5-dc90d42c378c\" (UID: \"c14dd981-8a33-4993-88c5-dc90d42c378c\") "
Jul 06 20:50:39.129496 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.129467    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14dd981-8a33-4993-88c5-dc90d42c378c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c14dd981-8a33-4993-88c5-dc90d42c378c" (UID: "c14dd981-8a33-4993-88c5-dc90d42c378c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:50:39.129568 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.129496    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c14dd981-8a33-4993-88c5-dc90d42c378c" (UID: "c14dd981-8a33-4993-88c5-dc90d42c378c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:50:39.129568 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.129536    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c14dd981-8a33-4993-88c5-dc90d42c378c" (UID: "c14dd981-8a33-4993-88c5-dc90d42c378c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:50:39.131355 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.131332    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14dd981-8a33-4993-88c5-dc90d42c378c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c14dd981-8a33-4993-88c5-dc90d42c378c" (UID: "c14dd981-8a33-4993-88c5-dc90d42c378c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:50:39.131446 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.131412    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-api-access-j7dhz" (OuterVolumeSpecName: "kube-api-access-j7dhz") pod "c14dd981-8a33-4993-88c5-dc90d42c378c" (UID: "c14dd981-8a33-4993-88c5-dc90d42c378c"). InnerVolumeSpecName "kube-api-access-j7dhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:50:39.230689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.230609    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dhz\" (UniqueName: \"kubernetes.io/projected/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-api-access-j7dhz\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:50:39.230689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.230636    2541 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:50:39.230689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.230651    2541 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c14dd981-8a33-4993-88c5-dc90d42c378c-proxy-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:50:39.230689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.230665    2541 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c14dd981-8a33-4993-88c5-dc90d42c378c-kserve-provision-location\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:50:39.230689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.230678    2541 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c14dd981-8a33-4993-88c5-dc90d42c378c-cabundle-cert\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:50:39.567555 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.567525    2541 generic.go:356] "Generic (PLEG): container finished" podID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerID="2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3" exitCode=0
Jul 06 20:50:39.567672 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.567605    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerDied","Data":"2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3"}
Jul 06 20:50:39.567744 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.567670    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679" event={"ID":"c14dd981-8a33-4993-88c5-dc90d42c378c","Type":"ContainerDied","Data":"94621c4f5d8d5ec4d7d9fbe9efab2f38b662489466bd3b6f569d149dc1ccdd99"}
Jul 06 20:50:39.567744 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.567618    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"
Jul 06 20:50:39.567744 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.567708    2541 scope.go:117] "RemoveContainer" containerID="2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3"
Jul 06 20:50:39.576892 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.576871    2541 scope.go:117] "RemoveContainer" containerID="8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88"
Jul 06 20:50:39.586966 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.586951    2541 scope.go:117] "RemoveContainer" containerID="eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc"
Jul 06 20:50:39.589203 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.589185    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"]
Jul 06 20:50:39.592193 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.592174    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8659b9c9df-jw679"]
Jul 06 20:50:39.595467 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.595446    2541 scope.go:117] "RemoveContainer" containerID="ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2"
Jul 06 20:50:39.602345 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.602321    2541 scope.go:117] "RemoveContainer" containerID="2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3"
Jul 06 20:50:39.602607 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:50:39.602586    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3\": container with ID starting with 2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3 not found: ID does not exist" containerID="2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3"
Jul 06 20:50:39.602658 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.602619    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3"} err="failed to get container status \"2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3\": rpc error: code = NotFound desc = could not find container \"2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3\": container with ID starting with 2d14d850ec97a9a45f5a40e8c6f4286b2d9e017842926f889a1046e9a4e941a3 not found: ID does not exist"
Jul 06 20:50:39.602658 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.602644    2541 scope.go:117] "RemoveContainer" containerID="8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88"
Jul 06 20:50:39.602907 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:50:39.602890    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88\": container with ID starting with 8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88 not found: ID does not exist" containerID="8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88"
Jul 06 20:50:39.602956 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.602912    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88"} err="failed to get container status \"8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88\": rpc error: code = NotFound desc = could not find container \"8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88\": container with ID starting with 8c9782d9f796b088bbf2f19007c0e5b8a274068dcd05f9edcff2ca60755a0a88 not found: ID does not exist"
Jul 06 20:50:39.602956 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.602927    2541 scope.go:117] "RemoveContainer" containerID="eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc"
Jul 06 20:50:39.603183 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:50:39.603166    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc\": container with ID starting with eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc not found: ID does not exist" containerID="eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc"
Jul 06 20:50:39.603242 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.603187    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc"} err="failed to get container status \"eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc\": rpc error: code = NotFound desc = could not find container \"eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc\": container with ID starting with eceeeb67fd7e8be7dc1476a290267a0c805da9c2c60c1818e816258717b3c9bc not found: ID does not exist"
Jul 06 20:50:39.603242 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.603201    2541 scope.go:117] "RemoveContainer" containerID="ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2"
Jul 06 20:50:39.603464 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:50:39.603438    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2\": container with ID starting with ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2 not found: ID does not exist" containerID="ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2"
Jul 06 20:50:39.603530 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:39.603473    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2"} err="failed to get container status \"ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2\": rpc error: code = NotFound desc = could not find container \"ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2\": container with ID starting with ff99b8dcef8e38bfca6af19badc82693eb41b20846f8d261afcdaeef2073fce2 not found: ID does not exist"
Jul 06 20:50:41.091829 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:41.091801    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" path="/var/lib/kubelet/pods/c14dd981-8a33-4993-88c5-dc90d42c378c/volumes"
Jul 06 20:50:42.474160 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:42.474113    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:50:42.474552 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:42.474461    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:50:52.474250 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:52.474203    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:50:52.474680 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:50:52.474600    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:02.473479 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:02.473437    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:51:02.473856 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:02.473838    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:12.473421 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:12.473375    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:51:12.473863 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:12.473704    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:22.474236 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:22.474209    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:51:22.474638 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:22.474395    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:51:34.026986 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.026954    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"]
Jul 06 20:51:34.027363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027306    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent"
Jul 06 20:51:34.027363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027317    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent"
Jul 06 20:51:34.027363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027326    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy"
Jul 06 20:51:34.027363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027332    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy"
Jul 06 20:51:34.027363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027346    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="storage-initializer"
Jul 06 20:51:34.027363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027351    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="storage-initializer"
Jul 06 20:51:34.027363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027358    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container"
Jul 06 20:51:34.027363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027363    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container"
Jul 06 20:51:34.027588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027418    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kserve-container"
Jul 06 20:51:34.027588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027426    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="agent"
Jul 06 20:51:34.027588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.027436    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14dd981-8a33-4993-88c5-dc90d42c378c" containerName="kube-rbac-proxy"
Jul 06 20:51:34.031687 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.031668    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.033976 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.033955    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 06 20:51:34.034145 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.034127    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 06 20:51:34.052080 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.052060    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"]
Jul 06 20:51:34.092122 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.092101    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"]
Jul 06 20:51:34.092333 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.092310    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" containerID="cri-o://cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf" gracePeriod=30
Jul 06 20:51:34.092401 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.092332    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy" containerID="cri-o://0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee" gracePeriod=30
Jul 06 20:51:34.092401 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.092314    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" containerID="cri-o://85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e" gracePeriod=30
Jul 06 20:51:34.094297 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.094270    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.094403 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.094331    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.094403 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.094362    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwgct\" (UniqueName: \"kubernetes.io/projected/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-api-access-zwgct\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.194904 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.194881    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.194991 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.194923    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.194991 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.194943    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwgct\" (UniqueName: \"kubernetes.io/projected/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-api-access-zwgct\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.195106 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:51:34.195032    2541 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 06 20:51:34.195147 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:51:34.195136    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-proxy-tls podName:48ce7d64-6e1e-4f5a-9e2e-66997b6856de nodeName:}" failed. No retries permitted until 2026-07-06 20:51:34.695109628 +0000 UTC m=+970.037012324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-wtm57" (UID: "48ce7d64-6e1e-4f5a-9e2e-66997b6856de") : secret "message-dumper-predictor-serving-cert" not found
Jul 06 20:51:34.195507 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.195490    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.202737 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.202712    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwgct\" (UniqueName: \"kubernetes.io/projected/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-api-access-zwgct\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.697832 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.697800    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.700252 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.700225    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wtm57\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:34.798726 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.798699    2541 generic.go:356] "Generic (PLEG): container finished" podID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerID="0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee" exitCode=2
Jul 06 20:51:34.798849 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.798778    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerDied","Data":"0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee"}
Jul 06 20:51:34.941611 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:34.941589    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:35.067138 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:35.067112    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"]
Jul 06 20:51:35.069495 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:51:35.069470    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ce7d64_6e1e_4f5a_9e2e_66997b6856de.slice/crio-dca401383484acb6db45dd83aa57ecb25180f476d572bd7c186bc8fcb7115d2b WatchSource:0}: Error finding container dca401383484acb6db45dd83aa57ecb25180f476d572bd7c186bc8fcb7115d2b: Status 404 returned error can't find the container with id dca401383484acb6db45dd83aa57ecb25180f476d572bd7c186bc8fcb7115d2b
Jul 06 20:51:35.071282 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:35.071268    2541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:51:35.804562 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:35.804525    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57" event={"ID":"48ce7d64-6e1e-4f5a-9e2e-66997b6856de","Type":"ContainerStarted","Data":"dca401383484acb6db45dd83aa57ecb25180f476d572bd7c186bc8fcb7115d2b"}
Jul 06 20:51:36.812035 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:36.811998    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57" event={"ID":"48ce7d64-6e1e-4f5a-9e2e-66997b6856de","Type":"ContainerStarted","Data":"60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824"}
Jul 06 20:51:36.812444 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:36.812086    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57" event={"ID":"48ce7d64-6e1e-4f5a-9e2e-66997b6856de","Type":"ContainerStarted","Data":"c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32"}
Jul 06 20:51:36.812444 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:36.812121    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:36.812444 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:36.812151    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:36.813809 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:36.813790    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:36.828839 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:36.828799    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57" podStartSLOduration=2.43523611 podStartE2EDuration="3.82878601s" totalImagesPullingTime="1.3935499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:51:36.826911466 +0000 UTC m=+972.168814182" watchObservedRunningTime="2026-07-06 20:51:36.82878601 +0000 UTC m=+972.170688725"
Jul 06 20:51:37.469868 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:37.469834    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 06 20:51:38.823111 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:38.823083    2541 generic.go:356] "Generic (PLEG): container finished" podID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerID="85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e" exitCode=0
Jul 06 20:51:38.823429 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:38.823156    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerDied","Data":"85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e"}
Jul 06 20:51:42.469205 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:42.469166    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 06 20:51:42.473494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:42.473468    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:51:42.473791 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:42.473769    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:43.827561 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:43.827539    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:51:47.469733 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:47.469642    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 06 20:51:47.470151 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:47.469807    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:51:52.469744 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:52.469708    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 06 20:51:52.473972 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:52.473948    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:51:52.474341 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:52.474319    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:51:54.084621 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.084584    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"]
Jul 06 20:51:54.088323 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.088307    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.090363 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.090340    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 06 20:51:54.090463 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.090376    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 06 20:51:54.097925 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.097905    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"]
Jul 06 20:51:54.131469 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.131445    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-proxy-tls\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.131556 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.131483    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.131556 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.131502    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kserve-provision-location\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.131632 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.131562    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-cabundle-cert\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.131632 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.131597    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztcrd\" (UniqueName: \"kubernetes.io/projected/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-api-access-ztcrd\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.232183 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.232159    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.232266 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.232189    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kserve-provision-location\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.232266 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.232212    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-cabundle-cert\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.232266 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.232247    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztcrd\" (UniqueName: \"kubernetes.io/projected/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-api-access-ztcrd\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.232405 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.232306    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-proxy-tls\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.232459 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:51:54.232409    2541 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 06 20:51:54.232512 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:51:54.232473    2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-proxy-tls podName:c72b45ae-cee2-4fa6-98cd-9e101dd7baad nodeName:}" failed. No retries permitted until 2026-07-06 20:51:54.732453193 +0000 UTC m=+990.074355903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-proxy-tls") pod "isvc-logger-predictor-8b4d65f94-tdltz" (UID: "c72b45ae-cee2-4fa6-98cd-9e101dd7baad") : secret "isvc-logger-predictor-serving-cert" not found
Jul 06 20:51:54.232588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.232567    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kserve-provision-location\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.232854 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.232835    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.232924 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.232882    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-cabundle-cert\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.240622 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.240598    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztcrd\" (UniqueName: \"kubernetes.io/projected/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-api-access-ztcrd\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.736265 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.736242    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-proxy-tls\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.738353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.738330    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-proxy-tls\") pod \"isvc-logger-predictor-8b4d65f94-tdltz\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:54.999435 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:54.999389    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:55.127968 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:55.127948    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"]
Jul 06 20:51:55.129927 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:51:55.129899    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72b45ae_cee2_4fa6_98cd_9e101dd7baad.slice/crio-1a8c90e2ffcbf90542dae18d35acc8fa264ccbf001fbdc5dc34526d3161f8920 WatchSource:0}: Error finding container 1a8c90e2ffcbf90542dae18d35acc8fa264ccbf001fbdc5dc34526d3161f8920: Status 404 returned error can't find the container with id 1a8c90e2ffcbf90542dae18d35acc8fa264ccbf001fbdc5dc34526d3161f8920
Jul 06 20:51:55.896824 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:55.896789    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerStarted","Data":"8e6f381a995f823579a7c93dabc45dd2ed8a2a24ecbd5259db6861cdbe9449b1"}
Jul 06 20:51:55.896974 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:55.896850    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerStarted","Data":"1a8c90e2ffcbf90542dae18d35acc8fa264ccbf001fbdc5dc34526d3161f8920"}
Jul 06 20:51:57.469410 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:57.469368    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 06 20:51:58.912242 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:58.912216    2541 generic.go:356] "Generic (PLEG): container finished" podID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerID="8e6f381a995f823579a7c93dabc45dd2ed8a2a24ecbd5259db6861cdbe9449b1" exitCode=0
Jul 06 20:51:58.912588 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:58.912264    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerDied","Data":"8e6f381a995f823579a7c93dabc45dd2ed8a2a24ecbd5259db6861cdbe9449b1"}
Jul 06 20:51:59.919321 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:59.919288    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerStarted","Data":"cd023d98b9d8665fb8ecae2e4eac6228f969d66c8536b12f85e420f057eb7ae1"}
Jul 06 20:51:59.919841 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:59.919350    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerStarted","Data":"40f1fb2a2194d9b5061ce27d38865551798d84397d5e930a657068cd2923d04f"}
Jul 06 20:51:59.919841 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:59.919373    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerStarted","Data":"98c45179cb016c37ef91f99c5199be998fb3465a8729ad1d6d41afec035a99f1"}
Jul 06 20:51:59.919841 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:59.919662    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:59.919841 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:59.919789    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:51:59.921111 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:59.921086    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:51:59.939650 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:51:59.939615    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podStartSLOduration=2.4673449339999998 podStartE2EDuration="5.939567638s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.472222704s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:51:59.938780013 +0000 UTC m=+995.280682729" watchObservedRunningTime="2026-07-06 20:51:59.939567638 +0000 UTC m=+995.281470353"
Jul 06 20:52:00.923543 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:00.923500    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:52:00.923892 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:00.923547    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:52:00.924684 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:00.924657    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:01.927807 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:01.927768    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:52:01.928253 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:01.928176    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:02.469801 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:02.469765    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 06 20:52:02.474203 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:02.474176    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:5000: connect: connection refused"
Jul 06 20:52:02.474286 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:02.474272    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:52:02.474554 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:02.474535    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:02.474615 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:02.474605    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:52:04.237454 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.237430    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:52:04.306291 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.306265    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld9tw\" (UniqueName: \"kubernetes.io/projected/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-api-access-ld9tw\") pod \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") "
Jul 06 20:52:04.306441 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.306302    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kserve-provision-location\") pod \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") "
Jul 06 20:52:04.306441 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.306347    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-rbac-proxy-sar-config\") pod \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") "
Jul 06 20:52:04.306441 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.306401    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9b7dcd-130e-4a50-bf1e-cd184790398d-proxy-tls\") pod \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") "
Jul 06 20:52:04.306441 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.306431    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-cabundle-cert\") pod \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\" (UID: \"4a9b7dcd-130e-4a50-bf1e-cd184790398d\") "
Jul 06 20:52:04.306659 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.306612    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a9b7dcd-130e-4a50-bf1e-cd184790398d" (UID: "4a9b7dcd-130e-4a50-bf1e-cd184790398d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:52:04.306770 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.306655    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a9b7dcd-130e-4a50-bf1e-cd184790398d" (UID: "4a9b7dcd-130e-4a50-bf1e-cd184790398d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:04.306833 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.306811    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a9b7dcd-130e-4a50-bf1e-cd184790398d" (UID: "4a9b7dcd-130e-4a50-bf1e-cd184790398d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:52:04.308295 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.308274    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-api-access-ld9tw" (OuterVolumeSpecName: "kube-api-access-ld9tw") pod "4a9b7dcd-130e-4a50-bf1e-cd184790398d" (UID: "4a9b7dcd-130e-4a50-bf1e-cd184790398d"). InnerVolumeSpecName "kube-api-access-ld9tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:52:04.308482 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.308463    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9b7dcd-130e-4a50-bf1e-cd184790398d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a9b7dcd-130e-4a50-bf1e-cd184790398d" (UID: "4a9b7dcd-130e-4a50-bf1e-cd184790398d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:52:04.407133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.407076    2541 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:52:04.407133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.407096    2541 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9b7dcd-130e-4a50-bf1e-cd184790398d-proxy-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:52:04.407133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.407106    2541 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9b7dcd-130e-4a50-bf1e-cd184790398d-cabundle-cert\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:52:04.407133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.407114    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld9tw\" (UniqueName: \"kubernetes.io/projected/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kube-api-access-ld9tw\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:52:04.407133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.407122    2541 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9b7dcd-130e-4a50-bf1e-cd184790398d-kserve-provision-location\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:52:04.944580 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.944559    2541 generic.go:356] "Generic (PLEG): container finished" podID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerID="cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf" exitCode=0
Jul 06 20:52:04.944670 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.944617    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"
Jul 06 20:52:04.944670 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.944638    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerDied","Data":"cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf"}
Jul 06 20:52:04.944739 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.944711    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh" event={"ID":"4a9b7dcd-130e-4a50-bf1e-cd184790398d","Type":"ContainerDied","Data":"dac37db2817f3fc014f8e752673d7f4cffff086f8573cb5cde432fe5c8d45e7c"}
Jul 06 20:52:04.944771 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.944756    2541 scope.go:117] "RemoveContainer" containerID="cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf"
Jul 06 20:52:04.952824 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.952806    2541 scope.go:117] "RemoveContainer" containerID="0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee"
Jul 06 20:52:04.959941 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.959927    2541 scope.go:117] "RemoveContainer" containerID="85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e"
Jul 06 20:52:04.964013 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.963992    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"]
Jul 06 20:52:04.967314 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.967299    2541 scope.go:117] "RemoveContainer" containerID="cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd"
Jul 06 20:52:04.967944 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.967925    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-744fd9775b-rmtdh"]
Jul 06 20:52:04.975765 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.975752    2541 scope.go:117] "RemoveContainer" containerID="cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf"
Jul 06 20:52:04.975998 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:52:04.975983    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf\": container with ID starting with cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf not found: ID does not exist" containerID="cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf"
Jul 06 20:52:04.976038 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.976007    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf"} err="failed to get container status \"cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf\": rpc error: code = NotFound desc = could not find container \"cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf\": container with ID starting with cadb9af4fcce030a88dd7b64967d20dcac80642606c611c9166c0633b95920bf not found: ID does not exist"
Jul 06 20:52:04.976038 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.976021    2541 scope.go:117] "RemoveContainer" containerID="0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee"
Jul 06 20:52:04.976232 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:52:04.976215    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee\": container with ID starting with 0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee not found: ID does not exist" containerID="0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee"
Jul 06 20:52:04.976264 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.976233    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee"} err="failed to get container status \"0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee\": rpc error: code = NotFound desc = could not find container \"0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee\": container with ID starting with 0d02c6d4426fb9a07b1574fc4d12708ef5801c06636de7e17d67f62d6aa5e8ee not found: ID does not exist"
Jul 06 20:52:04.976264 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.976247    2541 scope.go:117] "RemoveContainer" containerID="85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e"
Jul 06 20:52:04.976430 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:52:04.976416    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e\": container with ID starting with 85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e not found: ID does not exist" containerID="85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e"
Jul 06 20:52:04.976462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.976433    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e"} err="failed to get container status \"85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e\": rpc error: code = NotFound desc = could not find container \"85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e\": container with ID starting with 85a583056a721f2aa344720dc8c3444787ff064dab5ab238423fe7a1f7d2487e not found: ID does not exist"
Jul 06 20:52:04.976462 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.976443    2541 scope.go:117] "RemoveContainer" containerID="cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd"
Jul 06 20:52:04.976625 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:52:04.976608    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd\": container with ID starting with cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd not found: ID does not exist" containerID="cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd"
Jul 06 20:52:04.976668 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:04.976631    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd"} err="failed to get container status \"cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd\": rpc error: code = NotFound desc = could not find container \"cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd\": container with ID starting with cbf575c211033f4809b3230f153473eb803fbd39bada088a05771b4ba5a2d7dd not found: ID does not exist"
Jul 06 20:52:05.092827 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:05.092804    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" path="/var/lib/kubelet/pods/4a9b7dcd-130e-4a50-bf1e-cd184790398d/volumes"
Jul 06 20:52:06.931371 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:06.931343    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:52:06.931812 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:06.931777    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:52:06.932200 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:06.932177    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:16.931799 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:16.931760    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:52:16.932356 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:16.932329    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:26.932095 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:26.932057    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:52:26.932543 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:26.932433    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:36.932659 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:36.932618    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:52:36.933108 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:36.933086    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:46.932232 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:46.932184    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:52:46.932653 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:46.932633    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:52:56.932526 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:56.932479    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:52:56.933007 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:52:56.932982    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:53:06.932202 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:06.932176    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:53:06.932665 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:06.932425    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:53:19.095483 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.095455    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-wtm57_48ce7d64-6e1e-4f5a-9e2e-66997b6856de/kserve-container/0.log"
Jul 06 20:53:19.286748 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.286716    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"]
Jul 06 20:53:19.286979 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.286947    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" containerID="cri-o://98c45179cb016c37ef91f99c5199be998fb3465a8729ad1d6d41afec035a99f1" gracePeriod=30
Jul 06 20:53:19.286979 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.286970    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy" containerID="cri-o://40f1fb2a2194d9b5061ce27d38865551798d84397d5e930a657068cd2923d04f" gracePeriod=30
Jul 06 20:53:19.287179 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.286967    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" containerID="cri-o://cd023d98b9d8665fb8ecae2e4eac6228f969d66c8536b12f85e420f057eb7ae1" gracePeriod=30
Jul 06 20:53:19.313871 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.313846    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"]
Jul 06 20:53:19.314224 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314209    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="storage-initializer"
Jul 06 20:53:19.314275 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314227    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="storage-initializer"
Jul 06 20:53:19.314275 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314235    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy"
Jul 06 20:53:19.314275 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314240    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy"
Jul 06 20:53:19.314275 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314248    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container"
Jul 06 20:53:19.314275 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314253    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container"
Jul 06 20:53:19.314275 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314267    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent"
Jul 06 20:53:19.314275 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314271    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent"
Jul 06 20:53:19.314511 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314329    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="agent"
Jul 06 20:53:19.314511 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314339    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kserve-container"
Jul 06 20:53:19.314511 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.314346    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9b7dcd-130e-4a50-bf1e-cd184790398d" containerName="kube-rbac-proxy"
Jul 06 20:53:19.317740 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.317722    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.319715 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.319691    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 06 20:53:19.319813 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.319718    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 06 20:53:19.334992 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.334971    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"]
Jul 06 20:53:19.366307 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.366257    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"]
Jul 06 20:53:19.366448 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.366430    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57" podUID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerName="kserve-container" containerID="cri-o://c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32" gracePeriod=30
Jul 06 20:53:19.366557 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.366529    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57" podUID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerName="kube-rbac-proxy" containerID="cri-o://60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824" gracePeriod=30
Jul 06 20:53:19.432602 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.432576    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc2206-db86-4178-a91e-8d1173bd4928-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.432602 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.432603    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.432730 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.432621    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g94d5\" (UniqueName: \"kubernetes.io/projected/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-api-access-g94d5\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.432730 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.432639    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.432801 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.432737    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc2206-db86-4178-a91e-8d1173bd4928-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.533494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.533454    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc2206-db86-4178-a91e-8d1173bd4928-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.533611 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.533560    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc2206-db86-4178-a91e-8d1173bd4928-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.533611 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.533591    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.533725 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.533617    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g94d5\" (UniqueName: \"kubernetes.io/projected/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-api-access-g94d5\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.533725 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.533644    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.534130 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.533973    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc2206-db86-4178-a91e-8d1173bd4928-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.534511 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.534466    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.534910 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.534888    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.537353 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.537330    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc2206-db86-4178-a91e-8d1173bd4928-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.541033 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.541009    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g94d5\" (UniqueName: \"kubernetes.io/projected/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-api-access-g94d5\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.595442 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.595422    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:53:19.629684 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.629630    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:53:19.735556 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.735533    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwgct\" (UniqueName: \"kubernetes.io/projected/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-api-access-zwgct\") pod \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") "
Jul 06 20:53:19.735699 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.735583    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-proxy-tls\") pod \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") "
Jul 06 20:53:19.735699 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.735659    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-rbac-proxy-sar-config\") pod \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\" (UID: \"48ce7d64-6e1e-4f5a-9e2e-66997b6856de\") "
Jul 06 20:53:19.736110 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.736077    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48ce7d64-6e1e-4f5a-9e2e-66997b6856de" (UID: "48ce7d64-6e1e-4f5a-9e2e-66997b6856de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:53:19.737737 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.737709    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-api-access-zwgct" (OuterVolumeSpecName: "kube-api-access-zwgct") pod "48ce7d64-6e1e-4f5a-9e2e-66997b6856de" (UID: "48ce7d64-6e1e-4f5a-9e2e-66997b6856de"). InnerVolumeSpecName "kube-api-access-zwgct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:53:19.737818 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.737760    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48ce7d64-6e1e-4f5a-9e2e-66997b6856de" (UID: "48ce7d64-6e1e-4f5a-9e2e-66997b6856de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:53:19.756793 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.756770    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"]
Jul 06 20:53:19.758665 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:53:19.758643    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebc2206_db86_4178_a91e_8d1173bd4928.slice/crio-cc62910b046e485d035f5993be00fa02840eecf7c1e54cd0f6173e2d8187c55f WatchSource:0}: Error finding container cc62910b046e485d035f5993be00fa02840eecf7c1e54cd0f6173e2d8187c55f: Status 404 returned error can't find the container with id cc62910b046e485d035f5993be00fa02840eecf7c1e54cd0f6173e2d8187c55f
Jul 06 20:53:19.836853 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.836832    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwgct\" (UniqueName: \"kubernetes.io/projected/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-api-access-zwgct\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:53:19.836952 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.836858    2541 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-proxy-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:53:19.836952 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:19.836874    2541 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48ce7d64-6e1e-4f5a-9e2e-66997b6856de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:53:20.268476 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.268446    2541 generic.go:356] "Generic (PLEG): container finished" podID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerID="40f1fb2a2194d9b5061ce27d38865551798d84397d5e930a657068cd2923d04f" exitCode=2
Jul 06 20:53:20.268886 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.268518    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerDied","Data":"40f1fb2a2194d9b5061ce27d38865551798d84397d5e930a657068cd2923d04f"}
Jul 06 20:53:20.269847 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.269823    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" event={"ID":"1ebc2206-db86-4178-a91e-8d1173bd4928","Type":"ContainerStarted","Data":"a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f"}
Jul 06 20:53:20.269948 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.269878    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" event={"ID":"1ebc2206-db86-4178-a91e-8d1173bd4928","Type":"ContainerStarted","Data":"cc62910b046e485d035f5993be00fa02840eecf7c1e54cd0f6173e2d8187c55f"}
Jul 06 20:53:20.271385 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.271363    2541 generic.go:356] "Generic (PLEG): container finished" podID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerID="60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824" exitCode=2
Jul 06 20:53:20.271385 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.271382    2541 generic.go:356] "Generic (PLEG): container finished" podID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerID="c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32" exitCode=2
Jul 06 20:53:20.271496 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.271409    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"
Jul 06 20:53:20.271496 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.271416    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57" event={"ID":"48ce7d64-6e1e-4f5a-9e2e-66997b6856de","Type":"ContainerDied","Data":"60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824"}
Jul 06 20:53:20.271496 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.271458    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57" event={"ID":"48ce7d64-6e1e-4f5a-9e2e-66997b6856de","Type":"ContainerDied","Data":"c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32"}
Jul 06 20:53:20.271496 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.271477    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57" event={"ID":"48ce7d64-6e1e-4f5a-9e2e-66997b6856de","Type":"ContainerDied","Data":"dca401383484acb6db45dd83aa57ecb25180f476d572bd7c186bc8fcb7115d2b"}
Jul 06 20:53:20.271686 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.271502    2541 scope.go:117] "RemoveContainer" containerID="60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824"
Jul 06 20:53:20.279376 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.279360    2541 scope.go:117] "RemoveContainer" containerID="c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32"
Jul 06 20:53:20.286379 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.286363    2541 scope.go:117] "RemoveContainer" containerID="60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824"
Jul 06 20:53:20.286593 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:53:20.286576    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824\": container with ID starting with 60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824 not found: ID does not exist" containerID="60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824"
Jul 06 20:53:20.286638 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.286601    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824"} err="failed to get container status \"60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824\": rpc error: code = NotFound desc = could not find container \"60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824\": container with ID starting with 60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824 not found: ID does not exist"
Jul 06 20:53:20.286638 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.286617    2541 scope.go:117] "RemoveContainer" containerID="c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32"
Jul 06 20:53:20.286842 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:53:20.286824    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32\": container with ID starting with c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32 not found: ID does not exist" containerID="c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32"
Jul 06 20:53:20.286888 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.286845    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32"} err="failed to get container status \"c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32\": rpc error: code = NotFound desc = could not find container \"c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32\": container with ID starting with c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32 not found: ID does not exist"
Jul 06 20:53:20.286888 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.286857    2541 scope.go:117] "RemoveContainer" containerID="60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824"
Jul 06 20:53:20.287133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.287029    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824"} err="failed to get container status \"60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824\": rpc error: code = NotFound desc = could not find container \"60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824\": container with ID starting with 60bd7c95417be33f17fb9b3c9e61216c39a966feae81bd0ac908f45d0f74b824 not found: ID does not exist"
Jul 06 20:53:20.287133 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.287065    2541 scope.go:117] "RemoveContainer" containerID="c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32"
Jul 06 20:53:20.287292 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.287278    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32"} err="failed to get container status \"c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32\": rpc error: code = NotFound desc = could not find container \"c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32\": container with ID starting with c86262915d2b737508c3a70b2501233fc731173dfae3dc30216fe1cff7841e32 not found: ID does not exist"
Jul 06 20:53:20.304178 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.304161    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"]
Jul 06 20:53:20.306537 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:20.306514    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wtm57"]
Jul 06 20:53:21.092864 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:21.092830    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" path="/var/lib/kubelet/pods/48ce7d64-6e1e-4f5a-9e2e-66997b6856de/volumes"
Jul 06 20:53:21.927982 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:21.927935    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 06 20:53:23.287388 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:23.287363    2541 generic.go:356] "Generic (PLEG): container finished" podID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerID="98c45179cb016c37ef91f99c5199be998fb3465a8729ad1d6d41afec035a99f1" exitCode=0
Jul 06 20:53:23.287715 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:23.287436    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerDied","Data":"98c45179cb016c37ef91f99c5199be998fb3465a8729ad1d6d41afec035a99f1"}
Jul 06 20:53:26.927822 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:26.927782    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 06 20:53:26.932015 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:26.931991    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:53:26.932385 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:26.932360    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:53:31.927780 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:31.927738    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 06 20:53:31.928231 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:31.927840    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:53:36.927784 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:36.927748    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 06 20:53:36.932085 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:36.932027    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:53:36.932391 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:36.932369    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:53:40.361243 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:40.361211    2541 generic.go:356] "Generic (PLEG): container finished" podID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerID="a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f" exitCode=0
Jul 06 20:53:40.361531 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:40.361281    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" event={"ID":"1ebc2206-db86-4178-a91e-8d1173bd4928","Type":"ContainerDied","Data":"a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f"}
Jul 06 20:53:41.928419 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:41.928363    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 06 20:53:46.927836 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:46.927795    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 06 20:53:46.932163 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:46.932132    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 06 20:53:46.932307 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:46.932294    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:53:46.932653 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:46.932623    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 20:53:46.932790 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:46.932749    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:53:49.411983 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.411944    2541 generic.go:356] "Generic (PLEG): container finished" podID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerID="cd023d98b9d8665fb8ecae2e4eac6228f969d66c8536b12f85e420f057eb7ae1" exitCode=0
Jul 06 20:53:49.412549 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.412088    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerDied","Data":"cd023d98b9d8665fb8ecae2e4eac6228f969d66c8536b12f85e420f057eb7ae1"}
Jul 06 20:53:49.471281 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.471260    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:53:49.544551 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.544424    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-proxy-tls\") pod \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") "
Jul 06 20:53:49.544551 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.544467    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-rbac-proxy-sar-config\") pod \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") "
Jul 06 20:53:49.544551 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.544481    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-cabundle-cert\") pod \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") "
Jul 06 20:53:49.544551 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.544518    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kserve-provision-location\") pod \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") "
Jul 06 20:53:49.544551 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.544548    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztcrd\" (UniqueName: \"kubernetes.io/projected/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-api-access-ztcrd\") pod \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\" (UID: \"c72b45ae-cee2-4fa6-98cd-9e101dd7baad\") "
Jul 06 20:53:49.544884 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.544827    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c72b45ae-cee2-4fa6-98cd-9e101dd7baad" (UID: "c72b45ae-cee2-4fa6-98cd-9e101dd7baad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:53:49.544955 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.544840    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c72b45ae-cee2-4fa6-98cd-9e101dd7baad" (UID: "c72b45ae-cee2-4fa6-98cd-9e101dd7baad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:53:49.545003 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.544971    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c72b45ae-cee2-4fa6-98cd-9e101dd7baad" (UID: "c72b45ae-cee2-4fa6-98cd-9e101dd7baad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:53:49.547112 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.547087    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c72b45ae-cee2-4fa6-98cd-9e101dd7baad" (UID: "c72b45ae-cee2-4fa6-98cd-9e101dd7baad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:53:49.547250 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.547201    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-api-access-ztcrd" (OuterVolumeSpecName: "kube-api-access-ztcrd") pod "c72b45ae-cee2-4fa6-98cd-9e101dd7baad" (UID: "c72b45ae-cee2-4fa6-98cd-9e101dd7baad"). InnerVolumeSpecName "kube-api-access-ztcrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:53:49.645278 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.645246    2541 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kserve-provision-location\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:53:49.645278 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.645275    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztcrd\" (UniqueName: \"kubernetes.io/projected/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-api-access-ztcrd\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:53:49.645436 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.645290    2541 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-proxy-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:53:49.645436 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.645305    2541 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:53:49.645436 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:49.645317    2541 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b45ae-cee2-4fa6-98cd-9e101dd7baad-cabundle-cert\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:53:50.419829 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:50.419798    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"
Jul 06 20:53:50.419829 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:50.419818    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz" event={"ID":"c72b45ae-cee2-4fa6-98cd-9e101dd7baad","Type":"ContainerDied","Data":"1a8c90e2ffcbf90542dae18d35acc8fa264ccbf001fbdc5dc34526d3161f8920"}
Jul 06 20:53:50.420327 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:50.419901    2541 scope.go:117] "RemoveContainer" containerID="cd023d98b9d8665fb8ecae2e4eac6228f969d66c8536b12f85e420f057eb7ae1"
Jul 06 20:53:50.430010 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:50.429994    2541 scope.go:117] "RemoveContainer" containerID="40f1fb2a2194d9b5061ce27d38865551798d84397d5e930a657068cd2923d04f"
Jul 06 20:53:50.438212 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:50.438191    2541 scope.go:117] "RemoveContainer" containerID="98c45179cb016c37ef91f99c5199be998fb3465a8729ad1d6d41afec035a99f1"
Jul 06 20:53:50.442775 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:50.442730    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"]
Jul 06 20:53:50.447980 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:50.447964    2541 scope.go:117] "RemoveContainer" containerID="8e6f381a995f823579a7c93dabc45dd2ed8a2a24ecbd5259db6861cdbe9449b1"
Jul 06 20:53:50.448734 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:50.448715    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8b4d65f94-tdltz"]
Jul 06 20:53:51.094271 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:53:51.094235    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" path="/var/lib/kubelet/pods/c72b45ae-cee2-4fa6-98cd-9e101dd7baad/volumes"
Jul 06 20:55:46.951754 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:55:46.951719    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" event={"ID":"1ebc2206-db86-4178-a91e-8d1173bd4928","Type":"ContainerStarted","Data":"f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893"}
Jul 06 20:55:46.952194 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:55:46.951799    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" event={"ID":"1ebc2206-db86-4178-a91e-8d1173bd4928","Type":"ContainerStarted","Data":"30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c"}
Jul 06 20:55:46.952194 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:55:46.951889    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:55:46.989035 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:55:46.988991    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" podStartSLOduration=1.966321283 podStartE2EDuration="2m27.988978283s" totalImagesPullingTime="2m5.524410727s" totalInitContainerRuntime="20.498246273s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:55:46.987914297 +0000 UTC m=+1222.329817004" watchObservedRunningTime="2026-07-06 20:55:46.988978283 +0000 UTC m=+1222.330880998"
Jul 06 20:55:47.956151 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:55:47.956116    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:55:47.956939 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:55:47.956914    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:55:48.959922 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:55:48.959887    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:55:53.964853 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:55:53.964824    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:55:53.965398 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:55:53.965342    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 06 20:56:03.966178 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:03.966150    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:56:09.493663 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:09.493631    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"]
Jul 06 20:56:09.494145 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:09.493877    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kserve-container" containerID="cri-o://30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c" gracePeriod=30
Jul 06 20:56:09.494145 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:09.493966    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kube-rbac-proxy" containerID="cri-o://f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893" gracePeriod=30
Jul 06 20:56:10.050571 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.050541    2541 generic.go:356] "Generic (PLEG): container finished" podID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerID="f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893" exitCode=2
Jul 06 20:56:10.050684 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.050599    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" event={"ID":"1ebc2206-db86-4178-a91e-8d1173bd4928","Type":"ContainerDied","Data":"f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893"}
Jul 06 20:56:10.266419 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.266395    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:56:10.309015 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.308955    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-rbac-proxy-sar-config\") pod \"1ebc2206-db86-4178-a91e-8d1173bd4928\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") "
Jul 06 20:56:10.309015 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.308993    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g94d5\" (UniqueName: \"kubernetes.io/projected/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-api-access-g94d5\") pod \"1ebc2206-db86-4178-a91e-8d1173bd4928\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") "
Jul 06 20:56:10.309213 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.309017    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc2206-db86-4178-a91e-8d1173bd4928-proxy-tls\") pod \"1ebc2206-db86-4178-a91e-8d1173bd4928\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") "
Jul 06 20:56:10.309213 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.309064    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc2206-db86-4178-a91e-8d1173bd4928-kserve-provision-location\") pod \"1ebc2206-db86-4178-a91e-8d1173bd4928\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") "
Jul 06 20:56:10.309213 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.309143    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-cabundle-cert\") pod \"1ebc2206-db86-4178-a91e-8d1173bd4928\" (UID: \"1ebc2206-db86-4178-a91e-8d1173bd4928\") "
Jul 06 20:56:10.309610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.309404    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ebc2206-db86-4178-a91e-8d1173bd4928" (UID: "1ebc2206-db86-4178-a91e-8d1173bd4928"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:56:10.309610 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.309477    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ebc2206-db86-4178-a91e-8d1173bd4928" (UID: "1ebc2206-db86-4178-a91e-8d1173bd4928"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:56:10.311139 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.311110    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebc2206-db86-4178-a91e-8d1173bd4928-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ebc2206-db86-4178-a91e-8d1173bd4928" (UID: "1ebc2206-db86-4178-a91e-8d1173bd4928"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:56:10.311240 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.311209    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-api-access-g94d5" (OuterVolumeSpecName: "kube-api-access-g94d5") pod "1ebc2206-db86-4178-a91e-8d1173bd4928" (UID: "1ebc2206-db86-4178-a91e-8d1173bd4928"). InnerVolumeSpecName "kube-api-access-g94d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:56:10.313722 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.313702    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebc2206-db86-4178-a91e-8d1173bd4928-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ebc2206-db86-4178-a91e-8d1173bd4928" (UID: "1ebc2206-db86-4178-a91e-8d1173bd4928"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:56:10.409637 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.409620    2541 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:56:10.409637 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.409637    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g94d5\" (UniqueName: \"kubernetes.io/projected/1ebc2206-db86-4178-a91e-8d1173bd4928-kube-api-access-g94d5\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:56:10.409773 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.409647    2541 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebc2206-db86-4178-a91e-8d1173bd4928-proxy-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:56:10.409773 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.409655    2541 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebc2206-db86-4178-a91e-8d1173bd4928-kserve-provision-location\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:56:10.409773 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:10.409663    2541 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebc2206-db86-4178-a91e-8d1173bd4928-cabundle-cert\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:56:11.056009 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.055982    2541 generic.go:356] "Generic (PLEG): container finished" podID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerID="30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c" exitCode=0
Jul 06 20:56:11.056322 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.056059    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"
Jul 06 20:56:11.056322 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.056072    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" event={"ID":"1ebc2206-db86-4178-a91e-8d1173bd4928","Type":"ContainerDied","Data":"30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c"}
Jul 06 20:56:11.056322 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.056126    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz" event={"ID":"1ebc2206-db86-4178-a91e-8d1173bd4928","Type":"ContainerDied","Data":"cc62910b046e485d035f5993be00fa02840eecf7c1e54cd0f6173e2d8187c55f"}
Jul 06 20:56:11.056322 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.056152    2541 scope.go:117] "RemoveContainer" containerID="f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893"
Jul 06 20:56:11.066037 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.066012    2541 scope.go:117] "RemoveContainer" containerID="30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c"
Jul 06 20:56:11.073476 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.073460    2541 scope.go:117] "RemoveContainer" containerID="a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f"
Jul 06 20:56:11.080580 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.080561    2541 scope.go:117] "RemoveContainer" containerID="f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893"
Jul 06 20:56:11.080801 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.080784    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"]
Jul 06 20:56:11.080851 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:56:11.080807    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893\": container with ID starting with f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893 not found: ID does not exist" containerID="f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893"
Jul 06 20:56:11.080851 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.080829    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893"} err="failed to get container status \"f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893\": rpc error: code = NotFound desc = could not find container \"f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893\": container with ID starting with f59f789492492611b287b8744003976f0f5e0edc28fe368d7f4d22ac1b652893 not found: ID does not exist"
Jul 06 20:56:11.080851 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.080844    2541 scope.go:117] "RemoveContainer" containerID="30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c"
Jul 06 20:56:11.081119 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:56:11.081103    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c\": container with ID starting with 30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c not found: ID does not exist" containerID="30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c"
Jul 06 20:56:11.081190 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.081128    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c"} err="failed to get container status \"30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c\": rpc error: code = NotFound desc = could not find container \"30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c\": container with ID starting with 30f0b755240f33f473d12f0dfa755d523d94fa13f865ef586657fde4324d193c not found: ID does not exist"
Jul 06 20:56:11.081190 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.081147    2541 scope.go:117] "RemoveContainer" containerID="a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f"
Jul 06 20:56:11.081368 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:56:11.081352    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f\": container with ID starting with a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f not found: ID does not exist" containerID="a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f"
Jul 06 20:56:11.081408 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.081374    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f"} err="failed to get container status \"a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f\": rpc error: code = NotFound desc = could not find container \"a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f\": container with ID starting with a7bdb9e6a6953e9732ca8d072759e669f6b4dcdc41f4784b451c6dfdfa463a9f not found: ID does not exist"
Jul 06 20:56:11.084618 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.084599    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-6k8jz"]
Jul 06 20:56:11.091830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:11.091811    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" path="/var/lib/kubelet/pods/1ebc2206-db86-4178-a91e-8d1173bd4928/volumes"
Jul 06 20:56:14.795788 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.795756    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"]
Jul 06 20:56:14.796227 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796208    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerName="kserve-container"
Jul 06 20:56:14.796272 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796231    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerName="kserve-container"
Jul 06 20:56:14.796272 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796248    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kube-rbac-proxy"
Jul 06 20:56:14.796272 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796259    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kube-rbac-proxy"
Jul 06 20:56:14.796272 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796270    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerName="kube-rbac-proxy"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796278    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerName="kube-rbac-proxy"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796288    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="storage-initializer"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796296    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="storage-initializer"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796315    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796324    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796333    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kserve-container"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796341    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kserve-container"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796351    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="storage-initializer"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796359    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="storage-initializer"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796371    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796379    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796391    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent"
Jul 06 20:56:14.796406 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796398    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent"
Jul 06 20:56:14.796830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796491    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kube-rbac-proxy"
Jul 06 20:56:14.796830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796503    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="kserve-container"
Jul 06 20:56:14.796830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796513    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerName="kserve-container"
Jul 06 20:56:14.796830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796520    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kube-rbac-proxy"
Jul 06 20:56:14.796830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796528    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ce7d64-6e1e-4f5a-9e2e-66997b6856de" containerName="kube-rbac-proxy"
Jul 06 20:56:14.796830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796536    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72b45ae-cee2-4fa6-98cd-9e101dd7baad" containerName="agent"
Jul 06 20:56:14.796830 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.796544    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebc2206-db86-4178-a91e-8d1173bd4928" containerName="kserve-container"
Jul 06 20:56:14.819164 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.819133    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"]
Jul 06 20:56:14.819336 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.819208    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.821264 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.821230    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:56:14.821402 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.821326    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 20:56:14.821402 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.821326    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 20:56:14.821561 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.821547    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 06 20:56:14.821742 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.821720    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 20:56:14.821815 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.821721    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 06 20:56:14.822123 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.822086    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5qwfj\""
Jul 06 20:56:14.839332 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.839307    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.839460 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.839339    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzpq4\" (UniqueName: \"kubernetes.io/projected/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-api-access-kzpq4\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.839460 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.839356    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.839460 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.839399    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea97b6d4-48a3-4308-a103-2215547d06f6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.839460 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.839440    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea97b6d4-48a3-4308-a103-2215547d06f6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.940729 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.940697    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea97b6d4-48a3-4308-a103-2215547d06f6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.940914 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.940753    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.940914 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.940774    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzpq4\" (UniqueName: \"kubernetes.io/projected/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-api-access-kzpq4\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.940914 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.940789    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.940914 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.940807    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea97b6d4-48a3-4308-a103-2215547d06f6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.941371 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.941339    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea97b6d4-48a3-4308-a103-2215547d06f6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.941755 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.941570    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.941848 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.941780    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.943437 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.943414    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea97b6d4-48a3-4308-a103-2215547d06f6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:14.947990 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:14.947966    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzpq4\" (UniqueName: \"kubernetes.io/projected/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-api-access-kzpq4\") pod \"isvc-autogluon-v2-predictor-67b849d456-swmsq\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:15.129544 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:15.129469    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:15.298905 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:15.298784    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"]
Jul 06 20:56:16.078505 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:16.078474    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" event={"ID":"ea97b6d4-48a3-4308-a103-2215547d06f6","Type":"ContainerStarted","Data":"931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95"}
Jul 06 20:56:16.078880 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:16.078529    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" event={"ID":"ea97b6d4-48a3-4308-a103-2215547d06f6","Type":"ContainerStarted","Data":"f4d6a6782e9c77901d837a52cfaee0d2d189423e872600088dd9ceb0934bc0fc"}
Jul 06 20:56:45.200495 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:45.200468    2541 generic.go:356] "Generic (PLEG): container finished" podID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerID="931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95" exitCode=0
Jul 06 20:56:45.200880 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:45.200542    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" event={"ID":"ea97b6d4-48a3-4308-a103-2215547d06f6","Type":"ContainerDied","Data":"931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95"}
Jul 06 20:56:46.206684 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:46.206647    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" event={"ID":"ea97b6d4-48a3-4308-a103-2215547d06f6","Type":"ContainerStarted","Data":"5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1"}
Jul 06 20:56:46.207174 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:46.206755    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" event={"ID":"ea97b6d4-48a3-4308-a103-2215547d06f6","Type":"ContainerStarted","Data":"e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265"}
Jul 06 20:56:46.207174 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:46.206979    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:46.225294 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:46.225243    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" podStartSLOduration=3.075262737 podStartE2EDuration="32.225225439s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.149962702s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:56:46.224616768 +0000 UTC m=+1281.566519483" watchObservedRunningTime="2026-07-06 20:56:46.225225439 +0000 UTC m=+1281.567128157"
Jul 06 20:56:47.211063 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:47.211025    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:56:53.220464 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:56:53.220433    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:58:15.098067 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:15.097977    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:58:24.952322 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:24.952291    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"]
Jul 06 20:58:24.952788 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:24.952494    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="kserve-container" containerID="cri-o://e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265" gracePeriod=30
Jul 06 20:58:24.952788 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:24.952534    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="kube-rbac-proxy" containerID="cri-o://5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1" gracePeriod=30
Jul 06 20:58:25.090393 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.090359    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 06 20:58:25.593910 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.593880    2541 generic.go:356] "Generic (PLEG): container finished" podID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerID="5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1" exitCode=2
Jul 06 20:58:25.594019 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.593949    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" event={"ID":"ea97b6d4-48a3-4308-a103-2215547d06f6","Type":"ContainerDied","Data":"5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1"}
Jul 06 20:58:25.786468 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.786448    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:58:25.929743 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.929690    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-cabundle-cert\") pod \"ea97b6d4-48a3-4308-a103-2215547d06f6\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") "
Jul 06 20:58:25.929743 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.929724    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea97b6d4-48a3-4308-a103-2215547d06f6-kserve-provision-location\") pod \"ea97b6d4-48a3-4308-a103-2215547d06f6\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") "
Jul 06 20:58:25.929904 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.929751    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzpq4\" (UniqueName: \"kubernetes.io/projected/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-api-access-kzpq4\") pod \"ea97b6d4-48a3-4308-a103-2215547d06f6\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") "
Jul 06 20:58:25.929904 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.929790    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea97b6d4-48a3-4308-a103-2215547d06f6-proxy-tls\") pod \"ea97b6d4-48a3-4308-a103-2215547d06f6\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") "
Jul 06 20:58:25.929904 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.929809    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-rbac-proxy-sar-config\") pod \"ea97b6d4-48a3-4308-a103-2215547d06f6\" (UID: \"ea97b6d4-48a3-4308-a103-2215547d06f6\") "
Jul 06 20:58:25.930116 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.930027    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea97b6d4-48a3-4308-a103-2215547d06f6" (UID: "ea97b6d4-48a3-4308-a103-2215547d06f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:58:25.930242 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.930222    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea97b6d4-48a3-4308-a103-2215547d06f6" (UID: "ea97b6d4-48a3-4308-a103-2215547d06f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:58:25.931913 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.931887    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea97b6d4-48a3-4308-a103-2215547d06f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea97b6d4-48a3-4308-a103-2215547d06f6" (UID: "ea97b6d4-48a3-4308-a103-2215547d06f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:58:25.932012 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.931952    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-api-access-kzpq4" (OuterVolumeSpecName: "kube-api-access-kzpq4") pod "ea97b6d4-48a3-4308-a103-2215547d06f6" (UID: "ea97b6d4-48a3-4308-a103-2215547d06f6"). InnerVolumeSpecName "kube-api-access-kzpq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:58:25.934407 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:25.934389    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea97b6d4-48a3-4308-a103-2215547d06f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea97b6d4-48a3-4308-a103-2215547d06f6" (UID: "ea97b6d4-48a3-4308-a103-2215547d06f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:58:26.030888 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.030861    2541 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-cabundle-cert\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:58:26.030888 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.030887    2541 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea97b6d4-48a3-4308-a103-2215547d06f6-kserve-provision-location\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:58:26.031232 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.030901    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzpq4\" (UniqueName: \"kubernetes.io/projected/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-api-access-kzpq4\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:58:26.031232 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.030913    2541 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea97b6d4-48a3-4308-a103-2215547d06f6-proxy-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:58:26.031232 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.030928    2541 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea97b6d4-48a3-4308-a103-2215547d06f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 20:58:26.600731 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.600705    2541 generic.go:356] "Generic (PLEG): container finished" podID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerID="e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265" exitCode=0
Jul 06 20:58:26.600890 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.600751    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" event={"ID":"ea97b6d4-48a3-4308-a103-2215547d06f6","Type":"ContainerDied","Data":"e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265"}
Jul 06 20:58:26.600890 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.600765    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"
Jul 06 20:58:26.600890 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.600796    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq" event={"ID":"ea97b6d4-48a3-4308-a103-2215547d06f6","Type":"ContainerDied","Data":"f4d6a6782e9c77901d837a52cfaee0d2d189423e872600088dd9ceb0934bc0fc"}
Jul 06 20:58:26.600890 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.600821    2541 scope.go:117] "RemoveContainer" containerID="5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1"
Jul 06 20:58:26.609365 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.609347    2541 scope.go:117] "RemoveContainer" containerID="e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265"
Jul 06 20:58:26.616667 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.616654    2541 scope.go:117] "RemoveContainer" containerID="931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95"
Jul 06 20:58:26.621513 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.621494    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"]
Jul 06 20:58:26.623991 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.623972    2541 scope.go:117] "RemoveContainer" containerID="5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1"
Jul 06 20:58:26.624276 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:58:26.624259    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1\": container with ID starting with 5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1 not found: ID does not exist" containerID="5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1"
Jul 06 20:58:26.624340 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.624284    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1"} err="failed to get container status \"5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1\": rpc error: code = NotFound desc = could not find container \"5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1\": container with ID starting with 5eb0fb42b9b1f8e72e5b39a8650f3a8d8f6ec1034507fc4e3604ac876f4ef5b1 not found: ID does not exist"
Jul 06 20:58:26.624340 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.624299    2541 scope.go:117] "RemoveContainer" containerID="e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265"
Jul 06 20:58:26.624567 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:58:26.624548    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265\": container with ID starting with e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265 not found: ID does not exist" containerID="e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265"
Jul 06 20:58:26.624615 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.624573    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265"} err="failed to get container status \"e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265\": rpc error: code = NotFound desc = could not find container \"e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265\": container with ID starting with e06c6921d053b43903c21f871fb7ad0b3e2cfd7bbb307d7a33788e7dfa87f265 not found: ID does not exist"
Jul 06 20:58:26.624615 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.624589    2541 scope.go:117] "RemoveContainer" containerID="931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95"
Jul 06 20:58:26.624824 ip-10-0-134-39 kubenswrapper[2541]: E0706 20:58:26.624805    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95\": container with ID starting with 931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95 not found: ID does not exist" containerID="931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95"
Jul 06 20:58:26.624871 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.624821    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95"} err="failed to get container status \"931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95\": rpc error: code = NotFound desc = could not find container \"931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95\": container with ID starting with 931eab16b545f14ec5828694e9fc97136a6dda4d6b2a38cd15e3d4f1f2dd8b95 not found: ID does not exist"
Jul 06 20:58:26.627202 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:26.627182    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-swmsq"]
Jul 06 20:58:27.091494 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:27.091468    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" path="/var/lib/kubelet/pods/ea97b6d4-48a3-4308-a103-2215547d06f6/volumes"
Jul 06 20:58:30.099841 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.099816    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"]
Jul 06 20:58:30.100182 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.100104    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="kserve-container"
Jul 06 20:58:30.100182 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.100116    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="kserve-container"
Jul 06 20:58:30.100182 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.100139    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="storage-initializer"
Jul 06 20:58:30.100182 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.100144    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="storage-initializer"
Jul 06 20:58:30.100182 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.100149    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="kube-rbac-proxy"
Jul 06 20:58:30.100182 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.100155    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="kube-rbac-proxy"
Jul 06 20:58:30.100364 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.100199    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="kube-rbac-proxy"
Jul 06 20:58:30.100364 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.100205    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea97b6d4-48a3-4308-a103-2215547d06f6" containerName="kserve-container"
Jul 06 20:58:30.104983 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.104969    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.107162 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.107139    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 20:58:30.107651 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.107633    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 20:58:30.107816 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.107658    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 06 20:58:30.107890 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.107658    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 06 20:58:30.107890 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.107665    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 20:58:30.107999 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.107684    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5qwfj\""
Jul 06 20:58:30.107999 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.107741    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 20:58:30.112888 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.112865    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"]
Jul 06 20:58:30.160951 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.160928    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.161026 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.160968    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbp97\" (UniqueName: \"kubernetes.io/projected/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-api-access-kbp97\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.161026 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.160989    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.161157 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.161142    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.161211 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.161174    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.261766 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.261743    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.261861 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.261770    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbp97\" (UniqueName: \"kubernetes.io/projected/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-api-access-kbp97\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.261861 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.261794    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.261861 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.261823    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.262093 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.261924    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.262166 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.262138    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.262546 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.262526    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.262585 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.262548    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.264184 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.264168    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.269162 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.269140    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbp97\" (UniqueName: \"kubernetes.io/projected/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-api-access-kbp97\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.415645 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.415597    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:30.537689 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.537667    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"]
Jul 06 20:58:30.539504 ip-10-0-134-39 kubenswrapper[2541]: W0706 20:58:30.539479    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30acda3a_82ae_4c3f_b8e5_a593fe3ac757.slice/crio-a19239dd5eee1cd386007071dea39507a29f49277cabb11df978f3a55e3af77d WatchSource:0}: Error finding container a19239dd5eee1cd386007071dea39507a29f49277cabb11df978f3a55e3af77d: Status 404 returned error can't find the container with id a19239dd5eee1cd386007071dea39507a29f49277cabb11df978f3a55e3af77d
Jul 06 20:58:30.618342 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.618304    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" event={"ID":"30acda3a-82ae-4c3f-b8e5-a593fe3ac757","Type":"ContainerStarted","Data":"d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae"}
Jul 06 20:58:30.618454 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:30.618395    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" event={"ID":"30acda3a-82ae-4c3f-b8e5-a593fe3ac757","Type":"ContainerStarted","Data":"a19239dd5eee1cd386007071dea39507a29f49277cabb11df978f3a55e3af77d"}
Jul 06 20:58:35.640725 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:35.640653    2541 generic.go:356] "Generic (PLEG): container finished" podID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerID="d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae" exitCode=0
Jul 06 20:58:35.640725 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:35.640709    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" event={"ID":"30acda3a-82ae-4c3f-b8e5-a593fe3ac757","Type":"ContainerDied","Data":"d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae"}
Jul 06 20:58:36.654140 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:36.654087    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" event={"ID":"30acda3a-82ae-4c3f-b8e5-a593fe3ac757","Type":"ContainerStarted","Data":"e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c"}
Jul 06 20:58:36.654507 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:36.654190    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" event={"ID":"30acda3a-82ae-4c3f-b8e5-a593fe3ac757","Type":"ContainerStarted","Data":"b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db"}
Jul 06 20:58:36.654507 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:36.654385    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:36.654507 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:36.654434    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 20:58:36.674006 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:36.673958    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" podStartSLOduration=1.973835381 podStartE2EDuration="6.673946414s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.700111033s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:36.672256802 +0000 UTC m=+1392.014159515" watchObservedRunningTime="2026-07-06 20:58:36.673946414 +0000 UTC m=+1392.015849129"
Jul 06 20:58:42.665110 ip-10-0-134-39 kubenswrapper[2541]: I0706 20:58:42.665072    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 21:00:05.093672 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:05.093644    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 21:00:10.222835 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:10.222800    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"]
Jul 06 21:00:10.223332 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:10.223040    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="kube-rbac-proxy" containerID="cri-o://e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c" gracePeriod=30
Jul 06 21:00:10.223332 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:10.223039    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="kserve-container" containerID="cri-o://b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db" gracePeriod=30
Jul 06 21:00:10.961673 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:10.961654    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 21:00:11.002407 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.002378    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kserve-provision-location\") pod \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") "
Jul 06 21:00:11.002529 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.002412    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-cabundle-cert\") pod \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") "
Jul 06 21:00:11.002595 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.002520    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-rbac-proxy-sar-config\") pod \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") "
Jul 06 21:00:11.002595 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.002550    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbp97\" (UniqueName: \"kubernetes.io/projected/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-api-access-kbp97\") pod \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") "
Jul 06 21:00:11.002710 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.002616    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-proxy-tls\") pod \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\" (UID: \"30acda3a-82ae-4c3f-b8e5-a593fe3ac757\") "
Jul 06 21:00:11.002812 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.002779    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30acda3a-82ae-4c3f-b8e5-a593fe3ac757" (UID: "30acda3a-82ae-4c3f-b8e5-a593fe3ac757"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:00:11.002927 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.002864    2541 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-cabundle-cert\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:00:11.002927 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.002902    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30acda3a-82ae-4c3f-b8e5-a593fe3ac757" (UID: "30acda3a-82ae-4c3f-b8e5-a593fe3ac757"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:00:11.004633 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.004597    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-api-access-kbp97" (OuterVolumeSpecName: "kube-api-access-kbp97") pod "30acda3a-82ae-4c3f-b8e5-a593fe3ac757" (UID: "30acda3a-82ae-4c3f-b8e5-a593fe3ac757"). InnerVolumeSpecName "kube-api-access-kbp97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:00:11.004716 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.004671    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30acda3a-82ae-4c3f-b8e5-a593fe3ac757" (UID: "30acda3a-82ae-4c3f-b8e5-a593fe3ac757"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:00:11.006823 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.006804    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30acda3a-82ae-4c3f-b8e5-a593fe3ac757" (UID: "30acda3a-82ae-4c3f-b8e5-a593fe3ac757"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:00:11.064840 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.064788    2541 generic.go:356] "Generic (PLEG): container finished" podID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerID="e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c" exitCode=2
Jul 06 21:00:11.064840 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.064809    2541 generic.go:356] "Generic (PLEG): container finished" podID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerID="b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db" exitCode=0
Jul 06 21:00:11.064989 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.064845    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"
Jul 06 21:00:11.064989 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.064884    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" event={"ID":"30acda3a-82ae-4c3f-b8e5-a593fe3ac757","Type":"ContainerDied","Data":"e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c"}
Jul 06 21:00:11.064989 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.064954    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" event={"ID":"30acda3a-82ae-4c3f-b8e5-a593fe3ac757","Type":"ContainerDied","Data":"b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db"}
Jul 06 21:00:11.065156 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.064991    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g" event={"ID":"30acda3a-82ae-4c3f-b8e5-a593fe3ac757","Type":"ContainerDied","Data":"a19239dd5eee1cd386007071dea39507a29f49277cabb11df978f3a55e3af77d"}
Jul 06 21:00:11.065156 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.065019    2541 scope.go:117] "RemoveContainer" containerID="e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c"
Jul 06 21:00:11.073528 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.073511    2541 scope.go:117] "RemoveContainer" containerID="b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db"
Jul 06 21:00:11.082898 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.082855    2541 scope.go:117] "RemoveContainer" containerID="d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae"
Jul 06 21:00:11.084467 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.084449    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"]
Jul 06 21:00:11.090724 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.090703    2541 scope.go:117] "RemoveContainer" containerID="e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c"
Jul 06 21:00:11.091020 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:00:11.090995    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c\": container with ID starting with e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c not found: ID does not exist" containerID="e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c"
Jul 06 21:00:11.091180 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.091027    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c"} err="failed to get container status \"e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c\": rpc error: code = NotFound desc = could not find container \"e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c\": container with ID starting with e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c not found: ID does not exist"
Jul 06 21:00:11.091180 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.091068    2541 scope.go:117] "RemoveContainer" containerID="b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db"
Jul 06 21:00:11.091411 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:00:11.091392    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db\": container with ID starting with b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db not found: ID does not exist" containerID="b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db"
Jul 06 21:00:11.091479 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.091420    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db"} err="failed to get container status \"b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db\": rpc error: code = NotFound desc = could not find container \"b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db\": container with ID starting with b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db not found: ID does not exist"
Jul 06 21:00:11.091479 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.091441    2541 scope.go:117] "RemoveContainer" containerID="d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae"
Jul 06 21:00:11.091696 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:00:11.091676    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae\": container with ID starting with d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae not found: ID does not exist" containerID="d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae"
Jul 06 21:00:11.091766 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.091699    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae"} err="failed to get container status \"d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae\": rpc error: code = NotFound desc = could not find container \"d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae\": container with ID starting with d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae not found: ID does not exist"
Jul 06 21:00:11.091766 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.091712    2541 scope.go:117] "RemoveContainer" containerID="e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c"
Jul 06 21:00:11.091914 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.091898    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c"} err="failed to get container status \"e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c\": rpc error: code = NotFound desc = could not find container \"e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c\": container with ID starting with e0999c8c3ac634a11743be0c5b4711b635be6cc181064c9cd719a560df7be53c not found: ID does not exist"
Jul 06 21:00:11.091979 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.091914    2541 scope.go:117] "RemoveContainer" containerID="b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db"
Jul 06 21:00:11.092172 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.092151    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db"} err="failed to get container status \"b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db\": rpc error: code = NotFound desc = could not find container \"b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db\": container with ID starting with b172a98a52e3c69061640c2a9bf5317f61489b7fcc5a08501236edc3b545a2db not found: ID does not exist"
Jul 06 21:00:11.092242 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.092175    2541 scope.go:117] "RemoveContainer" containerID="d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae"
Jul 06 21:00:11.092421 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.092404    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae"} err="failed to get container status \"d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae\": rpc error: code = NotFound desc = could not find container \"d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae\": container with ID starting with d96f700727472c4688404e2f3a79183e2efc4ce45a62545d1d0cff0caaf341ae not found: ID does not exist"
Jul 06 21:00:11.093651 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.093636    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mp99g"]
Jul 06 21:00:11.104110 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.104094    2541 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:00:11.104168 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.104113    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbp97\" (UniqueName: \"kubernetes.io/projected/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kube-api-access-kbp97\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:00:11.104168 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.104122    2541 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-proxy-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:00:11.104168 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:11.104131    2541 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30acda3a-82ae-4c3f-b8e5-a593fe3ac757-kserve-provision-location\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:00:13.091727 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:13.091699    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" path="/var/lib/kubelet/pods/30acda3a-82ae-4c3f-b8e5-a593fe3ac757/volumes"
Jul 06 21:00:15.386511 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.386475    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"]
Jul 06 21:00:15.386892 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.386831    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="storage-initializer"
Jul 06 21:00:15.386892 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.386843    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="storage-initializer"
Jul 06 21:00:15.386892 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.386852    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="kube-rbac-proxy"
Jul 06 21:00:15.386892 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.386857    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="kube-rbac-proxy"
Jul 06 21:00:15.386892 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.386871    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="kserve-container"
Jul 06 21:00:15.386892 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.386877    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="kserve-container"
Jul 06 21:00:15.387147 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.386928    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="kserve-container"
Jul 06 21:00:15.387147 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.386937    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="30acda3a-82ae-4c3f-b8e5-a593fe3ac757" containerName="kube-rbac-proxy"
Jul 06 21:00:15.391619 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.391595    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.393539 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.393513    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:00:15.393539 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.393527    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5qwfj\""
Jul 06 21:00:15.393707 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.393531    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:00:15.393707 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.393628    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 06 21:00:15.394228 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.394209    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 06 21:00:15.394326 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.394269    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:00:15.394326 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.394295    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:00:15.403105 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.403086    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"]
Jul 06 21:00:15.432186 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.432167    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7kn8\" (UniqueName: \"kubernetes.io/projected/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-api-access-v7kn8\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.432279 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.432198    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.432279 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.432230    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.432279 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.432248    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.432382 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.432322    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.532877 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.532856    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.532952 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.532885    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.532952 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.532903    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.532952 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.532920    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.533103 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.532985    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7kn8\" (UniqueName: \"kubernetes.io/projected/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-api-access-v7kn8\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.533339 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.533322    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.533546 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.533525    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.533612 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.533575    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.535349 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.535329    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.539973 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.539952    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7kn8\" (UniqueName: \"kubernetes.io/projected/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-api-access-v7kn8\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.702117 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.702065    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:15.831825 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:15.831801    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"]
Jul 06 21:00:15.833833 ip-10-0-134-39 kubenswrapper[2541]: W0706 21:00:15.833805    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2fc12d_2da3_436e_84f4_43b96d6afd6f.slice/crio-c37a217536f3f1026e305d1496948629d5899813984f9d229bebb3b994ace2a5 WatchSource:0}: Error finding container c37a217536f3f1026e305d1496948629d5899813984f9d229bebb3b994ace2a5: Status 404 returned error can't find the container with id c37a217536f3f1026e305d1496948629d5899813984f9d229bebb3b994ace2a5
Jul 06 21:00:16.088374 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:16.088339    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" event={"ID":"4e2fc12d-2da3-436e-84f4-43b96d6afd6f","Type":"ContainerStarted","Data":"b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886"}
Jul 06 21:00:16.088532 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:16.088403    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" event={"ID":"4e2fc12d-2da3-436e-84f4-43b96d6afd6f","Type":"ContainerStarted","Data":"c37a217536f3f1026e305d1496948629d5899813984f9d229bebb3b994ace2a5"}
Jul 06 21:00:21.110816 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:21.110787    2541 generic.go:356] "Generic (PLEG): container finished" podID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerID="b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886" exitCode=0
Jul 06 21:00:21.111162 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:21.110865    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" event={"ID":"4e2fc12d-2da3-436e-84f4-43b96d6afd6f","Type":"ContainerDied","Data":"b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886"}
Jul 06 21:00:22.117812 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:22.117776    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" event={"ID":"4e2fc12d-2da3-436e-84f4-43b96d6afd6f","Type":"ContainerStarted","Data":"aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652"}
Jul 06 21:00:22.118298 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:22.117844    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" event={"ID":"4e2fc12d-2da3-436e-84f4-43b96d6afd6f","Type":"ContainerStarted","Data":"e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4"}
Jul 06 21:00:22.118298 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:22.118086    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:22.139123 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:22.139078    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" podStartSLOduration=2.593722279 podStartE2EDuration="7.139060372s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.545338093s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:00:22.137381655 +0000 UTC m=+1497.479284372" watchObservedRunningTime="2026-07-06 21:00:22.139060372 +0000 UTC m=+1497.480963080"
Jul 06 21:00:23.122329 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:23.122294    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:00:29.133149 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:00:29.133120    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:01:55.096025 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:55.095974    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:01:55.502113 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:55.502065    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"]
Jul 06 21:01:55.522935 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:55.522878    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="kserve-container" containerID="cri-o://e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4" gracePeriod=30
Jul 06 21:01:55.523111 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:55.522917    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="kube-rbac-proxy" containerID="cri-o://aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652" gracePeriod=30
Jul 06 21:01:56.270626 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.270600    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:01:56.455143 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.455063    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kserve-provision-location\") pod \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") "
Jul 06 21:01:56.455300 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.455178    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-rbac-proxy-sar-config\") pod \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") "
Jul 06 21:01:56.455300 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.455234    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-cabundle-cert\") pod \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") "
Jul 06 21:01:56.455300 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.455265    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-proxy-tls\") pod \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") "
Jul 06 21:01:56.455300 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.455292    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7kn8\" (UniqueName: \"kubernetes.io/projected/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-api-access-v7kn8\") pod \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\" (UID: \"4e2fc12d-2da3-436e-84f4-43b96d6afd6f\") "
Jul 06 21:01:56.455580 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.455553    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e2fc12d-2da3-436e-84f4-43b96d6afd6f" (UID: "4e2fc12d-2da3-436e-84f4-43b96d6afd6f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:01:56.455631 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.455598    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e2fc12d-2da3-436e-84f4-43b96d6afd6f" (UID: "4e2fc12d-2da3-436e-84f4-43b96d6afd6f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:01:56.457512 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.457482    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e2fc12d-2da3-436e-84f4-43b96d6afd6f" (UID: "4e2fc12d-2da3-436e-84f4-43b96d6afd6f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:01:56.457649 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.457624    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-api-access-v7kn8" (OuterVolumeSpecName: "kube-api-access-v7kn8") pod "4e2fc12d-2da3-436e-84f4-43b96d6afd6f" (UID: "4e2fc12d-2da3-436e-84f4-43b96d6afd6f"). InnerVolumeSpecName "kube-api-access-v7kn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:01:56.459330 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.459307    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e2fc12d-2da3-436e-84f4-43b96d6afd6f" (UID: "4e2fc12d-2da3-436e-84f4-43b96d6afd6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:01:56.528658 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.528627    2541 generic.go:356] "Generic (PLEG): container finished" podID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerID="aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652" exitCode=2
Jul 06 21:01:56.528658 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.528651    2541 generic.go:356] "Generic (PLEG): container finished" podID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerID="e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4" exitCode=0
Jul 06 21:01:56.528841 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.528681    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"
Jul 06 21:01:56.528841 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.528717    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" event={"ID":"4e2fc12d-2da3-436e-84f4-43b96d6afd6f","Type":"ContainerDied","Data":"aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652"}
Jul 06 21:01:56.528841 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.528790    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" event={"ID":"4e2fc12d-2da3-436e-84f4-43b96d6afd6f","Type":"ContainerDied","Data":"e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4"}
Jul 06 21:01:56.528841 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.528825    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj" event={"ID":"4e2fc12d-2da3-436e-84f4-43b96d6afd6f","Type":"ContainerDied","Data":"c37a217536f3f1026e305d1496948629d5899813984f9d229bebb3b994ace2a5"}
Jul 06 21:01:56.529007 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.528862    2541 scope.go:117] "RemoveContainer" containerID="aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652"
Jul 06 21:01:56.538273 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.538259    2541 scope.go:117] "RemoveContainer" containerID="e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4"
Jul 06 21:01:56.545426 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.545411    2541 scope.go:117] "RemoveContainer" containerID="b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886"
Jul 06 21:01:56.550463 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.550443    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"]
Jul 06 21:01:56.553403 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.553382    2541 scope.go:117] "RemoveContainer" containerID="aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652"
Jul 06 21:01:56.553656 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:01:56.553639    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652\": container with ID starting with aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652 not found: ID does not exist" containerID="aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652"
Jul 06 21:01:56.553747 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.553664    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652"} err="failed to get container status \"aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652\": rpc error: code = NotFound desc = could not find container \"aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652\": container with ID starting with aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652 not found: ID does not exist"
Jul 06 21:01:56.553747 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.553681    2541 scope.go:117] "RemoveContainer" containerID="e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4"
Jul 06 21:01:56.553922 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:01:56.553908    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4\": container with ID starting with e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4 not found: ID does not exist" containerID="e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4"
Jul 06 21:01:56.553965 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.553937    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4"} err="failed to get container status \"e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4\": rpc error: code = NotFound desc = could not find container \"e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4\": container with ID starting with e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4 not found: ID does not exist"
Jul 06 21:01:56.553965 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.553950    2541 scope.go:117] "RemoveContainer" containerID="b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886"
Jul 06 21:01:56.554223 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:01:56.554205    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886\": container with ID starting with b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886 not found: ID does not exist" containerID="b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886"
Jul 06 21:01:56.554284 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.554229    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886"} err="failed to get container status \"b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886\": rpc error: code = NotFound desc = could not find container \"b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886\": container with ID starting with b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886 not found: ID does not exist"
Jul 06 21:01:56.554284 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.554243    2541 scope.go:117] "RemoveContainer" containerID="aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652"
Jul 06 21:01:56.554383 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.554364    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-sxldj"]
Jul 06 21:01:56.554468 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.554448    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652"} err="failed to get container status \"aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652\": rpc error: code = NotFound desc = could not find container \"aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652\": container with ID starting with aebbe6164af9d6f07fcd3b812c4df6ac3dc733b3907e48b47290663d965fb652 not found: ID does not exist"
Jul 06 21:01:56.554507 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.554470    2541 scope.go:117] "RemoveContainer" containerID="e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4"
Jul 06 21:01:56.554707 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.554688    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4"} err="failed to get container status \"e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4\": rpc error: code = NotFound desc = could not find container \"e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4\": container with ID starting with e4a64ad73064338ecaa593f8ba564869369407801096faaf288aa95fd98f0bd4 not found: ID does not exist"
Jul 06 21:01:56.554778 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.554710    2541 scope.go:117] "RemoveContainer" containerID="b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886"
Jul 06 21:01:56.554926 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.554909    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886"} err="failed to get container status \"b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886\": rpc error: code = NotFound desc = could not find container \"b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886\": container with ID starting with b32371ea4356176656ac7da47d138d6a775e6f7378d8f536bc00737459abf886 not found: ID does not exist"
Jul 06 21:01:56.556202 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.556186    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7kn8\" (UniqueName: \"kubernetes.io/projected/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-api-access-v7kn8\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:01:56.556248 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.556206    2541 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kserve-provision-location\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:01:56.556248 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.556215    2541 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:01:56.556248 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.556224    2541 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-cabundle-cert\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:01:56.556248 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:56.556232    2541 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e2fc12d-2da3-436e-84f4-43b96d6afd6f-proxy-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:01:57.093124 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:01:57.093091    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" path="/var/lib/kubelet/pods/4e2fc12d-2da3-436e-84f4-43b96d6afd6f/volumes"
Jul 06 21:02:00.704150 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.704122    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"]
Jul 06 21:02:00.704502 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.704416    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="kserve-container"
Jul 06 21:02:00.704502 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.704426    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="kserve-container"
Jul 06 21:02:00.704502 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.704438    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="kube-rbac-proxy"
Jul 06 21:02:00.704502 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.704443    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="kube-rbac-proxy"
Jul 06 21:02:00.704502 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.704452    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="storage-initializer"
Jul 06 21:02:00.704502 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.704458    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="storage-initializer"
Jul 06 21:02:00.704695 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.704508    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="kserve-container"
Jul 06 21:02:00.704695 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.704516    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2fc12d-2da3-436e-84f4-43b96d6afd6f" containerName="kube-rbac-proxy"
Jul 06 21:02:00.707801 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.707780    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.709610 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.709584    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 06 21:02:00.709761 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.709742    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 06 21:02:00.709827 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.709777    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5qwfj\""
Jul 06 21:02:00.709881 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.709828    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 06 21:02:00.709937 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.709907    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 06 21:02:00.710248 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.710228    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 06 21:02:00.710248 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.710245    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 06 21:02:00.718148 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.718129    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"]
Jul 06 21:02:00.887082 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.887032    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2pc\" (UniqueName: \"kubernetes.io/projected/b7b309a5-656c-49da-949c-404a7a43ea13-kube-api-access-nt2pc\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.887221 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.887097    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7b309a5-656c-49da-949c-404a7a43ea13-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.887221 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.887136    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.887221 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.887191    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b309a5-656c-49da-949c-404a7a43ea13-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.887363 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.887251    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.987913 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.987888    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.988032 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.987921    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b309a5-656c-49da-949c-404a7a43ea13-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.988032 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.987973    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.988032 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.988000    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2pc\" (UniqueName: \"kubernetes.io/projected/b7b309a5-656c-49da-949c-404a7a43ea13-kube-api-access-nt2pc\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.988032 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.988030    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7b309a5-656c-49da-949c-404a7a43ea13-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.988498 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.988475    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7b309a5-656c-49da-949c-404a7a43ea13-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.988656 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.988636    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.988722 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.988675    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.990564 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.990546    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b309a5-656c-49da-949c-404a7a43ea13-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:00.995585 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:00.995560    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2pc\" (UniqueName: \"kubernetes.io/projected/b7b309a5-656c-49da-949c-404a7a43ea13-kube-api-access-nt2pc\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:01.019445 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:01.019423    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:01.147124 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:01.147103    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"]
Jul 06 21:02:01.150574 ip-10-0-134-39 kubenswrapper[2541]: W0706 21:02:01.150545    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b309a5_656c_49da_949c_404a7a43ea13.slice/crio-6c1a119680a0ddb811c6f25720d35b736031040fbd2e4c338163d82e2d4e9faf WatchSource:0}: Error finding container 6c1a119680a0ddb811c6f25720d35b736031040fbd2e4c338163d82e2d4e9faf: Status 404 returned error can't find the container with id 6c1a119680a0ddb811c6f25720d35b736031040fbd2e4c338163d82e2d4e9faf
Jul 06 21:02:01.554133 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:01.554097    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" event={"ID":"b7b309a5-656c-49da-949c-404a7a43ea13","Type":"ContainerStarted","Data":"9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0"}
Jul 06 21:02:01.554295 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:01.554165    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" event={"ID":"b7b309a5-656c-49da-949c-404a7a43ea13","Type":"ContainerStarted","Data":"6c1a119680a0ddb811c6f25720d35b736031040fbd2e4c338163d82e2d4e9faf"}
Jul 06 21:02:06.580771 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:06.580733    2541 generic.go:356] "Generic (PLEG): container finished" podID="b7b309a5-656c-49da-949c-404a7a43ea13" containerID="9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0" exitCode=0
Jul 06 21:02:06.581171 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:06.580808    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" event={"ID":"b7b309a5-656c-49da-949c-404a7a43ea13","Type":"ContainerDied","Data":"9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0"}
Jul 06 21:02:07.588483 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:07.588452    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" event={"ID":"b7b309a5-656c-49da-949c-404a7a43ea13","Type":"ContainerStarted","Data":"a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404"}
Jul 06 21:02:07.588845 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:07.588513    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" event={"ID":"b7b309a5-656c-49da-949c-404a7a43ea13","Type":"ContainerStarted","Data":"772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9"}
Jul 06 21:02:07.588845 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:07.588701    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:07.608840 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:07.608793    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" podStartSLOduration=3.214773377 podStartE2EDuration="7.608778146s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.394004769s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:02:07.607314323 +0000 UTC m=+1602.949217043" watchObservedRunningTime="2026-07-06 21:02:07.608778146 +0000 UTC m=+1602.950680864"
Jul 06 21:02:08.593560 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:08.593531    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:08.594753 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:08.594718    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 06 21:02:09.597817 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:09.597772    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 06 21:02:14.603721 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:14.603692    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:14.604247 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:14.604214    2541 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 06 21:02:24.605270 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:24.605179    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:30.728444 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:30.728413    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"]
Jul 06 21:02:30.728922 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:30.728642    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kserve-container" containerID="cri-o://772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9" gracePeriod=30
Jul 06 21:02:30.728922 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:30.728677    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kube-rbac-proxy" containerID="cri-o://a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404" gracePeriod=30
Jul 06 21:02:31.570637 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.570598    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:31.619623 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.619553    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b309a5-656c-49da-949c-404a7a43ea13-proxy-tls\") pod \"b7b309a5-656c-49da-949c-404a7a43ea13\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") "
Jul 06 21:02:31.619623 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.619593    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt2pc\" (UniqueName: \"kubernetes.io/projected/b7b309a5-656c-49da-949c-404a7a43ea13-kube-api-access-nt2pc\") pod \"b7b309a5-656c-49da-949c-404a7a43ea13\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") "
Jul 06 21:02:31.619623 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.619611    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-cabundle-cert\") pod \"b7b309a5-656c-49da-949c-404a7a43ea13\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") "
Jul 06 21:02:31.619879 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.619639    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7b309a5-656c-49da-949c-404a7a43ea13-kserve-provision-location\") pod \"b7b309a5-656c-49da-949c-404a7a43ea13\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") "
Jul 06 21:02:31.619879 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.619663    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-kube-rbac-proxy-sar-config\") pod \"b7b309a5-656c-49da-949c-404a7a43ea13\" (UID: \"b7b309a5-656c-49da-949c-404a7a43ea13\") "
Jul 06 21:02:31.620032 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.619995    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b7b309a5-656c-49da-949c-404a7a43ea13" (UID: "b7b309a5-656c-49da-949c-404a7a43ea13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:02:31.620120 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.620060    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b7b309a5-656c-49da-949c-404a7a43ea13" (UID: "b7b309a5-656c-49da-949c-404a7a43ea13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 21:02:31.620548 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.620523    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b309a5-656c-49da-949c-404a7a43ea13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b7b309a5-656c-49da-949c-404a7a43ea13" (UID: "b7b309a5-656c-49da-949c-404a7a43ea13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:02:31.621791 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.621771    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b309a5-656c-49da-949c-404a7a43ea13-kube-api-access-nt2pc" (OuterVolumeSpecName: "kube-api-access-nt2pc") pod "b7b309a5-656c-49da-949c-404a7a43ea13" (UID: "b7b309a5-656c-49da-949c-404a7a43ea13"). InnerVolumeSpecName "kube-api-access-nt2pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:02:31.621881 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.621812    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b309a5-656c-49da-949c-404a7a43ea13-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7b309a5-656c-49da-949c-404a7a43ea13" (UID: "b7b309a5-656c-49da-949c-404a7a43ea13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 21:02:31.701765 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.701734    2541 generic.go:356] "Generic (PLEG): container finished" podID="b7b309a5-656c-49da-949c-404a7a43ea13" containerID="a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404" exitCode=2
Jul 06 21:02:31.701765 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.701760    2541 generic.go:356] "Generic (PLEG): container finished" podID="b7b309a5-656c-49da-949c-404a7a43ea13" containerID="772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9" exitCode=0
Jul 06 21:02:31.701921 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.701785    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"
Jul 06 21:02:31.701921 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.701819    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" event={"ID":"b7b309a5-656c-49da-949c-404a7a43ea13","Type":"ContainerDied","Data":"a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404"}
Jul 06 21:02:31.701921 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.701873    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" event={"ID":"b7b309a5-656c-49da-949c-404a7a43ea13","Type":"ContainerDied","Data":"772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9"}
Jul 06 21:02:31.701921 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.701892    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm" event={"ID":"b7b309a5-656c-49da-949c-404a7a43ea13","Type":"ContainerDied","Data":"6c1a119680a0ddb811c6f25720d35b736031040fbd2e4c338163d82e2d4e9faf"}
Jul 06 21:02:31.701921 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.701916    2541 scope.go:117] "RemoveContainer" containerID="a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404"
Jul 06 21:02:31.710571 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.710555    2541 scope.go:117] "RemoveContainer" containerID="772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9"
Jul 06 21:02:31.717944 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.717923    2541 scope.go:117] "RemoveContainer" containerID="9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0"
Jul 06 21:02:31.720240 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.720221    2541 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7b309a5-656c-49da-949c-404a7a43ea13-proxy-tls\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:02:31.720412 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.720395    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt2pc\" (UniqueName: \"kubernetes.io/projected/b7b309a5-656c-49da-949c-404a7a43ea13-kube-api-access-nt2pc\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:02:31.720717 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.720629    2541 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-cabundle-cert\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:02:31.720717 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.720650    2541 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7b309a5-656c-49da-949c-404a7a43ea13-kserve-provision-location\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:02:31.720717 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.720677    2541 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7b309a5-656c-49da-949c-404a7a43ea13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:02:31.722537 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.722518    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"]
Jul 06 21:02:31.726470 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.726452    2541 scope.go:117] "RemoveContainer" containerID="a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404"
Jul 06 21:02:31.726759 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:02:31.726734    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404\": container with ID starting with a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404 not found: ID does not exist" containerID="a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404"
Jul 06 21:02:31.726847 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.726767    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404"} err="failed to get container status \"a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404\": rpc error: code = NotFound desc = could not find container \"a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404\": container with ID starting with a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404 not found: ID does not exist"
Jul 06 21:02:31.726847 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.726785    2541 scope.go:117] "RemoveContainer" containerID="772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9"
Jul 06 21:02:31.727021 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727003    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnhsm"]
Jul 06 21:02:31.727099 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:02:31.727036    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9\": container with ID starting with 772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9 not found: ID does not exist" containerID="772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9"
Jul 06 21:02:31.727099 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727076    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9"} err="failed to get container status \"772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9\": rpc error: code = NotFound desc = could not find container \"772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9\": container with ID starting with 772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9 not found: ID does not exist"
Jul 06 21:02:31.727099 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727091    2541 scope.go:117] "RemoveContainer" containerID="9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0"
Jul 06 21:02:31.727357 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:02:31.727341    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0\": container with ID starting with 9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0 not found: ID does not exist" containerID="9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0"
Jul 06 21:02:31.727408 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727362    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0"} err="failed to get container status \"9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0\": rpc error: code = NotFound desc = could not find container \"9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0\": container with ID starting with 9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0 not found: ID does not exist"
Jul 06 21:02:31.727408 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727377    2541 scope.go:117] "RemoveContainer" containerID="a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404"
Jul 06 21:02:31.727598 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727575    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404"} err="failed to get container status \"a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404\": rpc error: code = NotFound desc = could not find container \"a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404\": container with ID starting with a69144f085cc1e3523ff910473d1e3a0fea925aaf811a898581c43b6fb396404 not found: ID does not exist"
Jul 06 21:02:31.727642 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727599    2541 scope.go:117] "RemoveContainer" containerID="772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9"
Jul 06 21:02:31.727800 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727783    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9"} err="failed to get container status \"772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9\": rpc error: code = NotFound desc = could not find container \"772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9\": container with ID starting with 772ddc65691b1003c92d0c638c6c11675bcc3f3acb700c6d80773cb43298a8a9 not found: ID does not exist"
Jul 06 21:02:31.727853 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727799    2541 scope.go:117] "RemoveContainer" containerID="9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0"
Jul 06 21:02:31.727984 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:31.727969    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0"} err="failed to get container status \"9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0\": rpc error: code = NotFound desc = could not find container \"9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0\": container with ID starting with 9c81797c2ece787906eb1810ab9ea543c17c90175aa921b6a664aaa8eede4ae0 not found: ID does not exist"
Jul 06 21:02:33.092805 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:33.092762    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" path="/var/lib/kubelet/pods/b7b309a5-656c-49da-949c-404a7a43ea13/volumes"
Jul 06 21:02:37.002313 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.002277    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8j5ml/must-gather-24hdf"]
Jul 06 21:02:37.002886 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.002860    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kserve-container"
Jul 06 21:02:37.003006 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.002888    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kserve-container"
Jul 06 21:02:37.003006 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.002906    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kube-rbac-proxy"
Jul 06 21:02:37.003006 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.002915    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kube-rbac-proxy"
Jul 06 21:02:37.003006 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.002941    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="storage-initializer"
Jul 06 21:02:37.003006 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.002951    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="storage-initializer"
Jul 06 21:02:37.003308 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.003072    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kube-rbac-proxy"
Jul 06 21:02:37.003308 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.003089    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b309a5-656c-49da-949c-404a7a43ea13" containerName="kserve-container"
Jul 06 21:02:37.008305 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.008280    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:02:37.010435 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.010404    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8j5ml\"/\"openshift-service-ca.crt\""
Jul 06 21:02:37.011171 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.011142    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8j5ml\"/\"kube-root-ca.crt\""
Jul 06 21:02:37.011429 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.011410    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8j5ml\"/\"default-dockercfg-24z86\""
Jul 06 21:02:37.011544 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.011493    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8j5ml/must-gather-24hdf"]
Jul 06 21:02:37.059320 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.059287    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2mc\" (UniqueName: \"kubernetes.io/projected/5af8dae6-e210-41f1-b2a0-509c09a25450-kube-api-access-gc2mc\") pod \"must-gather-24hdf\" (UID: \"5af8dae6-e210-41f1-b2a0-509c09a25450\") " pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:02:37.059448 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.059354    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5af8dae6-e210-41f1-b2a0-509c09a25450-must-gather-output\") pod \"must-gather-24hdf\" (UID: \"5af8dae6-e210-41f1-b2a0-509c09a25450\") " pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:02:37.159861 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.159829    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5af8dae6-e210-41f1-b2a0-509c09a25450-must-gather-output\") pod \"must-gather-24hdf\" (UID: \"5af8dae6-e210-41f1-b2a0-509c09a25450\") " pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:02:37.159981 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.159913    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2mc\" (UniqueName: \"kubernetes.io/projected/5af8dae6-e210-41f1-b2a0-509c09a25450-kube-api-access-gc2mc\") pod \"must-gather-24hdf\" (UID: \"5af8dae6-e210-41f1-b2a0-509c09a25450\") " pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:02:37.160230 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.160209    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5af8dae6-e210-41f1-b2a0-509c09a25450-must-gather-output\") pod \"must-gather-24hdf\" (UID: \"5af8dae6-e210-41f1-b2a0-509c09a25450\") " pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:02:37.167379 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.167352    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2mc\" (UniqueName: \"kubernetes.io/projected/5af8dae6-e210-41f1-b2a0-509c09a25450-kube-api-access-gc2mc\") pod \"must-gather-24hdf\" (UID: \"5af8dae6-e210-41f1-b2a0-509c09a25450\") " pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:02:37.328123 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.328062    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:02:37.451152 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.451130    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8j5ml/must-gather-24hdf"]
Jul 06 21:02:37.453451 ip-10-0-134-39 kubenswrapper[2541]: W0706 21:02:37.453418    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5af8dae6_e210_41f1_b2a0_509c09a25450.slice/crio-4d0ed052b0c8f5d654a749be45b5afb92d996a829848b928408ffd5c54167e88 WatchSource:0}: Error finding container 4d0ed052b0c8f5d654a749be45b5afb92d996a829848b928408ffd5c54167e88: Status 404 returned error can't find the container with id 4d0ed052b0c8f5d654a749be45b5afb92d996a829848b928408ffd5c54167e88
Jul 06 21:02:37.455651 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.455631    2541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 21:02:37.728591 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:37.728561    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8j5ml/must-gather-24hdf" event={"ID":"5af8dae6-e210-41f1-b2a0-509c09a25450","Type":"ContainerStarted","Data":"4d0ed052b0c8f5d654a749be45b5afb92d996a829848b928408ffd5c54167e88"}
Jul 06 21:02:43.775172 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:43.775093    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8j5ml/must-gather-24hdf" event={"ID":"5af8dae6-e210-41f1-b2a0-509c09a25450","Type":"ContainerStarted","Data":"2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a"}
Jul 06 21:02:43.775172 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:43.775148    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8j5ml/must-gather-24hdf" event={"ID":"5af8dae6-e210-41f1-b2a0-509c09a25450","Type":"ContainerStarted","Data":"c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef"}
Jul 06 21:02:43.789159 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:02:43.789103    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8j5ml/must-gather-24hdf" podStartSLOduration=1.7223465550000001 podStartE2EDuration="7.789083001s" totalImagesPullingTime="6.066736446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:02:43.788889982 +0000 UTC m=+1639.130792702" watchObservedRunningTime="2026-07-06 21:02:43.789083001 +0000 UTC m=+1639.130985718"
Jul 06 21:03:03.872473 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:03.872393    2541 generic.go:356] "Generic (PLEG): container finished" podID="5af8dae6-e210-41f1-b2a0-509c09a25450" containerID="c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef" exitCode=0
Jul 06 21:03:03.872473 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:03.872468    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8j5ml/must-gather-24hdf" event={"ID":"5af8dae6-e210-41f1-b2a0-509c09a25450","Type":"ContainerDied","Data":"c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef"}
Jul 06 21:03:03.872897 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:03.872776    2541 scope.go:117] "RemoveContainer" containerID="c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef"
Jul 06 21:03:04.561765 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:04.561728    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8j5ml_must-gather-24hdf_5af8dae6-e210-41f1-b2a0-509c09a25450/gather/0.log"
Jul 06 21:03:07.845922 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:07.845892    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p68f6_12201c1c-c44f-4ce1-8d69-228ae73bd969/global-pull-secret-syncer/0.log"
Jul 06 21:03:08.006682 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:08.006656    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-466g2_2ee4fbe5-4e1b-46c7-8f1b-526edcfdad59/konnectivity-agent/0.log"
Jul 06 21:03:08.082833 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:08.082811    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-39.ec2.internal_487ce3ac867cb6a29534fc53f0fccb26/haproxy/0.log"
Jul 06 21:03:09.925289 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:09.925254    2541 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-8j5ml/must-gather-24hdf"]
Jul 06 21:03:09.925691 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:09.925422    2541 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-8j5ml/must-gather-24hdf" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" containerName="copy" containerID="cri-o://2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a" gracePeriod=2
Jul 06 21:03:09.929145 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:09.929115    2541 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-8j5ml/must-gather-24hdf"]
Jul 06 21:03:10.155784 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.155762    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8j5ml_must-gather-24hdf_5af8dae6-e210-41f1-b2a0-509c09a25450/copy/0.log"
Jul 06 21:03:10.158291 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.158274    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:03:10.160199 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:03:10.160175    2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-24hdf\" is forbidden: User \"system:node:ip-10-0-134-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8j5ml\": no relationship found between node 'ip-10-0-134-39.ec2.internal' and this object" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:03:10.340123 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.340097    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc2mc\" (UniqueName: \"kubernetes.io/projected/5af8dae6-e210-41f1-b2a0-509c09a25450-kube-api-access-gc2mc\") pod \"5af8dae6-e210-41f1-b2a0-509c09a25450\" (UID: \"5af8dae6-e210-41f1-b2a0-509c09a25450\") "
Jul 06 21:03:10.340266 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.340176    2541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5af8dae6-e210-41f1-b2a0-509c09a25450-must-gather-output\") pod \"5af8dae6-e210-41f1-b2a0-509c09a25450\" (UID: \"5af8dae6-e210-41f1-b2a0-509c09a25450\") "
Jul 06 21:03:10.341514 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.341488    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af8dae6-e210-41f1-b2a0-509c09a25450-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "5af8dae6-e210-41f1-b2a0-509c09a25450" (UID: "5af8dae6-e210-41f1-b2a0-509c09a25450"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 21:03:10.342423 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.342401    2541 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af8dae6-e210-41f1-b2a0-509c09a25450-kube-api-access-gc2mc" (OuterVolumeSpecName: "kube-api-access-gc2mc") pod "5af8dae6-e210-41f1-b2a0-509c09a25450" (UID: "5af8dae6-e210-41f1-b2a0-509c09a25450"). InnerVolumeSpecName "kube-api-access-gc2mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 21:03:10.441615 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.441590    2541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc2mc\" (UniqueName: \"kubernetes.io/projected/5af8dae6-e210-41f1-b2a0-509c09a25450-kube-api-access-gc2mc\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:03:10.441615 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.441610    2541 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5af8dae6-e210-41f1-b2a0-509c09a25450-must-gather-output\") on node \"ip-10-0-134-39.ec2.internal\" DevicePath \"\""
Jul 06 21:03:10.903395 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.903366    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-8j5ml_must-gather-24hdf_5af8dae6-e210-41f1-b2a0-509c09a25450/copy/0.log"
Jul 06 21:03:10.903734 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.903712    2541 generic.go:356] "Generic (PLEG): container finished" podID="5af8dae6-e210-41f1-b2a0-509c09a25450" containerID="2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a" exitCode=143
Jul 06 21:03:10.903885 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.903776    2541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:03:10.903885 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.903789    2541 scope.go:117] "RemoveContainer" containerID="2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a"
Jul 06 21:03:10.905887 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:03:10.905850    2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-24hdf\" is forbidden: User \"system:node:ip-10-0-134-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8j5ml\": no relationship found between node 'ip-10-0-134-39.ec2.internal' and this object" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:03:10.912234 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.912216    2541 scope.go:117] "RemoveContainer" containerID="c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef"
Jul 06 21:03:10.913257 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:03:10.913231    2541 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-24hdf\" is forbidden: User \"system:node:ip-10-0-134-39.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-8j5ml\": no relationship found between node 'ip-10-0-134-39.ec2.internal' and this object" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" pod="openshift-must-gather-8j5ml/must-gather-24hdf"
Jul 06 21:03:10.923416 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.923399    2541 scope.go:117] "RemoveContainer" containerID="2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a"
Jul 06 21:03:10.923671 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:03:10.923653    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a\": container with ID starting with 2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a not found: ID does not exist" containerID="2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a"
Jul 06 21:03:10.923721 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.923679    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a"} err="failed to get container status \"2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a\": rpc error: code = NotFound desc = could not find container \"2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a\": container with ID starting with 2d718f8d577b03d437ac546eff66adfdde4de9fc026c3362c6add6fdd0ca7d2a not found: ID does not exist"
Jul 06 21:03:10.923721 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.923696    2541 scope.go:117] "RemoveContainer" containerID="c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef"
Jul 06 21:03:10.923908 ip-10-0-134-39 kubenswrapper[2541]: E0706 21:03:10.923893    2541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef\": container with ID starting with c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef not found: ID does not exist" containerID="c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef"
Jul 06 21:03:10.923959 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:10.923913    2541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef"} err="failed to get container status \"c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef\": rpc error: code = NotFound desc = could not find container \"c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef\": container with ID starting with c59ecc9c19b4b951b5d8bfa720676774e43aedf03fda3610a10711b931bb62ef not found: ID does not exist"
Jul 06 21:03:11.092179 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:11.092149    2541 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" path="/var/lib/kubelet/pods/5af8dae6-e210-41f1-b2a0-509c09a25450/volumes"
Jul 06 21:03:12.287507 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.287482    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-qtl8n_71beb5de-c8d6-4dec-b5ac-8fc153f78270/cluster-monitoring-operator/0.log"
Jul 06 21:03:12.441012 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.440983    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-279vp_2c319108-a62c-40fe-a619-0ca069679212/monitoring-plugin/0.log"
Jul 06 21:03:12.544103 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.544006    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7l4gn_4dbbef56-95f7-4188-a285-d4474048b1b9/node-exporter/0.log"
Jul 06 21:03:12.567999 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.567975    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7l4gn_4dbbef56-95f7-4188-a285-d4474048b1b9/kube-rbac-proxy/0.log"
Jul 06 21:03:12.588111 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.588092    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7l4gn_4dbbef56-95f7-4188-a285-d4474048b1b9/init-textfile/0.log"
Jul 06 21:03:12.704146 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.704121    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7stxg_62736c9f-e9ac-47bb-883c-df7a4bc52ea0/kube-rbac-proxy-main/0.log"
Jul 06 21:03:12.726436 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.726410    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7stxg_62736c9f-e9ac-47bb-883c-df7a4bc52ea0/kube-rbac-proxy-self/0.log"
Jul 06 21:03:12.746951 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.746931    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7stxg_62736c9f-e9ac-47bb-883c-df7a4bc52ea0/openshift-state-metrics/0.log"
Jul 06 21:03:12.792770 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.792750    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c63b9af9-5191-4b42-af74-7104806be410/prometheus/0.log"
Jul 06 21:03:12.816956 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.816907    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c63b9af9-5191-4b42-af74-7104806be410/config-reloader/0.log"
Jul 06 21:03:12.853078 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.853037    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c63b9af9-5191-4b42-af74-7104806be410/thanos-sidecar/0.log"
Jul 06 21:03:12.877597 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.877579    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c63b9af9-5191-4b42-af74-7104806be410/kube-rbac-proxy-web/0.log"
Jul 06 21:03:12.898889 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.898871    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c63b9af9-5191-4b42-af74-7104806be410/kube-rbac-proxy/0.log"
Jul 06 21:03:12.924429 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.924414    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c63b9af9-5191-4b42-af74-7104806be410/kube-rbac-proxy-thanos/0.log"
Jul 06 21:03:12.947173 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.947129    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c63b9af9-5191-4b42-af74-7104806be410/init-config-reloader/0.log"
Jul 06 21:03:12.971558 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.971538    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-5vvz4_cd2afd06-71af-4154-98cf-711daa3d6ca1/prometheus-operator/0.log"
Jul 06 21:03:12.990884 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:12.990864    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-5vvz4_cd2afd06-71af-4154-98cf-711daa3d6ca1/kube-rbac-proxy/0.log"
Jul 06 21:03:13.013833 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:13.013813    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-zfqg2_ce7ce0b2-dbcc-455d-b760-d518524d02a6/prometheus-operator-admission-webhook/0.log"
Jul 06 21:03:14.568156 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:14.568126    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-c6w28_56f05cee-4e8c-49e7-a8f8-0c957946f881/networking-console-plugin/0.log"
Jul 06 21:03:16.008676 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.008650    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-f2959_546c165d-b822-40b8-8fae-7299137e126d/volume-data-source-validator/0.log"
Jul 06 21:03:16.194516 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.194486    2541 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"]
Jul 06 21:03:16.194788 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.194777    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" containerName="copy"
Jul 06 21:03:16.194832 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.194789    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" containerName="copy"
Jul 06 21:03:16.194832 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.194814    2541 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" containerName="gather"
Jul 06 21:03:16.194832 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.194820    2541 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" containerName="gather"
Jul 06 21:03:16.194930 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.194867    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" containerName="copy"
Jul 06 21:03:16.194930 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.194875    2541 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af8dae6-e210-41f1-b2a0-509c09a25450" containerName="gather"
Jul 06 21:03:16.199862 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.199841    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.202140 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.202118    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qqdbp\"/\"kube-root-ca.crt\""
Jul 06 21:03:16.202658 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.202644    2541 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qqdbp\"/\"default-dockercfg-h4br4\""
Jul 06 21:03:16.202658 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.202649    2541 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qqdbp\"/\"openshift-service-ca.crt\""
Jul 06 21:03:16.206634 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.206554    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"]
Jul 06 21:03:16.385627 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.385550    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-podres\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.385627 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.385583    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7b5\" (UniqueName: \"kubernetes.io/projected/e89e0751-b00f-4987-9947-b196228bd06c-kube-api-access-zf7b5\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.385627 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.385616    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-proc\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.385900 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.385747    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-sys\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.385900 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.385780    2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-lib-modules\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.486974 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.486943    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-sys\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.486974 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.486975    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-lib-modules\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.487240 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.486998    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-podres\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.487240 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.487099    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-sys\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.487240 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.487231    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7b5\" (UniqueName: \"kubernetes.io/projected/e89e0751-b00f-4987-9947-b196228bd06c-kube-api-access-zf7b5\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.487240 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.487234    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-lib-modules\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.487406 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.487230    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-podres\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.487406 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.487283    2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-proc\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.487406 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.487364    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e89e0751-b00f-4987-9947-b196228bd06c-proc\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.495069 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.495035    2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7b5\" (UniqueName: \"kubernetes.io/projected/e89e0751-b00f-4987-9947-b196228bd06c-kube-api-access-zf7b5\") pod \"perf-node-gather-daemonset-6jvmj\" (UID: \"e89e0751-b00f-4987-9947-b196228bd06c\") " pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.509974 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.509949    2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.635469 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.635294    2541 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"]
Jul 06 21:03:16.637726 ip-10-0-134-39 kubenswrapper[2541]: W0706 21:03:16.637660    2541 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode89e0751_b00f_4987_9947_b196228bd06c.slice/crio-aca771042f0639e9826fbb7e45364cd0e538f0a246d1cb39d4a9e1f48084d60b WatchSource:0}: Error finding container aca771042f0639e9826fbb7e45364cd0e538f0a246d1cb39d4a9e1f48084d60b: Status 404 returned error can't find the container with id aca771042f0639e9826fbb7e45364cd0e538f0a246d1cb39d4a9e1f48084d60b
Jul 06 21:03:16.931991 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.931899    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj" event={"ID":"e89e0751-b00f-4987-9947-b196228bd06c","Type":"ContainerStarted","Data":"e321dd2084f26b1d2641b2788aded544177d431e4be159bb7b01d5be34819221"}
Jul 06 21:03:16.931991 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.931951    2541 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj" event={"ID":"e89e0751-b00f-4987-9947-b196228bd06c","Type":"ContainerStarted","Data":"aca771042f0639e9826fbb7e45364cd0e538f0a246d1cb39d4a9e1f48084d60b"}
Jul 06 21:03:16.932219 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.932013    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:16.949207 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.949160    2541 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj" podStartSLOduration=0.949145617 podStartE2EDuration="949.145617ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:03:16.947371015 +0000 UTC m=+1672.289273732" watchObservedRunningTime="2026-07-06 21:03:16.949145617 +0000 UTC m=+1672.291048335"
Jul 06 21:03:16.967392 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.967371    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7fbsp_22e65afc-1f02-48a2-b7ff-9213eb6cbff7/dns/0.log"
Jul 06 21:03:16.989677 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:16.989627    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7fbsp_22e65afc-1f02-48a2-b7ff-9213eb6cbff7/kube-rbac-proxy/0.log"
Jul 06 21:03:17.052243 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:17.052201    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-784f2_73332827-2d68-4a90-ba2c-d3ed85704330/dns-node-resolver/0.log"
Jul 06 21:03:17.663495 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:17.663462    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vz5nr_4f3513df-fc03-42e5-86fc-bedae9f21fc0/node-ca/0.log"
Jul 06 21:03:18.855268 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:18.855236    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-55922_3721ac03-bf4f-477d-864a-f34cc9b9895c/serve-healthcheck-canary/0.log"
Jul 06 21:03:19.395939 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:19.395912    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mskws_60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b/kube-rbac-proxy/0.log"
Jul 06 21:03:19.418237 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:19.418212    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mskws_60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b/exporter/0.log"
Jul 06 21:03:19.437182 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:19.437155    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mskws_60b4bbf9-ebfd-4771-b60f-6b85a1b53c5b/extractor/0.log"
Jul 06 21:03:21.525747 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:21.525719    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-mhmm6_aa1bf996-1c87-4fac-adf7-a42b2c6ed4dd/manager/0.log"
Jul 06 21:03:21.556057 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:21.556025    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-t4ndd_10e8ceda-967f-43e4-9cf1-b78a6f78e683/s3-init/0.log"
Jul 06 21:03:21.570172 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:21.570155    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-t4ndd_10e8ceda-967f-43e4-9cf1-b78a6f78e683/download-hf-model/0.log"
Jul 06 21:03:21.595457 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:21.595422    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6tg6v_88d29ed2-92fe-4192-96b6-20b4326cc754/s3-tls-init-custom/0.log"
Jul 06 21:03:21.620853 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:21.620832    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-6tg6v_88d29ed2-92fe-4192-96b6-20b4326cc754/download-hf-model/0.log"
Jul 06 21:03:21.650240 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:21.650218    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-6d4kf_55b448e7-eae9-4325-8431-6133280eb6e9/s3-tls-init-serving/0.log"
Jul 06 21:03:21.666505 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:21.666487    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-6d4kf_55b448e7-eae9-4325-8431-6133280eb6e9/download-hf-model/0.log"
Jul 06 21:03:21.690243 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:21.690220    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-nv9cx_b5de7e6a-8427-4b18-a51d-669e3ba61fee/seaweedfs/0.log"
Jul 06 21:03:22.947249 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:22.947215    2541 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qqdbp/perf-node-gather-daemonset-6jvmj"
Jul 06 21:03:25.531924 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:25.531897    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-dbbjb_2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3/migrator/0.log"
Jul 06 21:03:25.551353 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:25.551331    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-dbbjb_2e427c32-d1e4-4be7-a443-0ba6f0a5a5c3/graceful-termination/0.log"
Jul 06 21:03:27.041553 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.041232    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t7wgp_448af9b4-632e-4e75-991a-f1a5138b9a52/kube-multus-additional-cni-plugins/0.log"
Jul 06 21:03:27.064624 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.064605    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t7wgp_448af9b4-632e-4e75-991a-f1a5138b9a52/egress-router-binary-copy/0.log"
Jul 06 21:03:27.096518 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.096462    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t7wgp_448af9b4-632e-4e75-991a-f1a5138b9a52/cni-plugins/0.log"
Jul 06 21:03:27.118194 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.118174    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t7wgp_448af9b4-632e-4e75-991a-f1a5138b9a52/bond-cni-plugin/0.log"
Jul 06 21:03:27.139443 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.139424    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t7wgp_448af9b4-632e-4e75-991a-f1a5138b9a52/routeoverride-cni/0.log"
Jul 06 21:03:27.160736 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.160716    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t7wgp_448af9b4-632e-4e75-991a-f1a5138b9a52/whereabouts-cni-bincopy/0.log"
Jul 06 21:03:27.181934 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.181914    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t7wgp_448af9b4-632e-4e75-991a-f1a5138b9a52/whereabouts-cni/0.log"
Jul 06 21:03:27.216240 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.216223    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fh5b5_9f19dafe-c8e6-4053-9d18-542a67692a26/kube-multus/0.log"
Jul 06 21:03:27.287573 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.287551    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4zbdm_4df480a7-e4bb-46d9-87d6-e07815d4f87b/network-metrics-daemon/0.log"
Jul 06 21:03:27.305142 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:27.305121    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4zbdm_4df480a7-e4bb-46d9-87d6-e07815d4f87b/kube-rbac-proxy/0.log"
Jul 06 21:03:28.105362 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:28.105333    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpn7x_40b3a754-3e30-4941-8978-ffe85a952f3a/ovn-controller/0.log"
Jul 06 21:03:28.121336 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:28.121311    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpn7x_40b3a754-3e30-4941-8978-ffe85a952f3a/ovn-acl-logging/0.log"
Jul 06 21:03:28.158941 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:28.158920    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpn7x_40b3a754-3e30-4941-8978-ffe85a952f3a/kube-rbac-proxy-node/0.log"
Jul 06 21:03:28.176850 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:28.176828    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpn7x_40b3a754-3e30-4941-8978-ffe85a952f3a/kube-rbac-proxy-ovn-metrics/0.log"
Jul 06 21:03:28.194400 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:28.194382    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpn7x_40b3a754-3e30-4941-8978-ffe85a952f3a/northd/0.log"
Jul 06 21:03:28.212708 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:28.212691    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpn7x_40b3a754-3e30-4941-8978-ffe85a952f3a/nbdb/0.log"
Jul 06 21:03:28.231688 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:28.231668    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpn7x_40b3a754-3e30-4941-8978-ffe85a952f3a/sbdb/0.log"
Jul 06 21:03:28.267285 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:28.267264    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gpn7x_40b3a754-3e30-4941-8978-ffe85a952f3a/ovnkube-controller/0.log"
Jul 06 21:03:29.566691 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:29.566661    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-989f8b8dc-zk2lg_08af6ed3-abe2-4380-9dd0-ea07dceadabc/check-endpoints/0.log"
Jul 06 21:03:29.588842 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:29.588818    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2vksd_a2f06fd8-d8d0-4169-9fa1-dbded13ed0cb/network-check-target-container/0.log"
Jul 06 21:03:30.525623 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:30.525591    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-w4qb2_821107a9-9583-4e63-869d-bff90d64af15/iptables-alerter/0.log"
Jul 06 21:03:31.167842 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:31.167811    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-r89gp_6ffdc1c3-89a0-4ac3-8c58-6125740d752e/tuned/0.log"
Jul 06 21:03:32.802135 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:32.802105    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-bnkn6_f45164fc-fdfa-41e7-a574-9b75551b0408/cluster-samples-operator/0.log"
Jul 06 21:03:32.817742 ip-10-0-134-39 kubenswrapper[2541]: I0706 21:03:32.817722    2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-bnkn6_f45164fc-fdfa-41e7-a574-9b75551b0408/cluster-samples-operator-watch/0.log"