Jul 13 18:47:58.450809 ip-10-0-130-96 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 13 18:47:58.450820 ip-10-0-130-96 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 13 18:47:58.450827 ip-10-0-130-96 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 13 18:47:58.451064 ip-10-0-130-96 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 13 18:48:08.698408 ip-10-0-130-96 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 13 18:48:08.698428 ip-10-0-130-96 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 82fc1b98a32f44dfad77893b1cd99aac --
Jul 13 18:50:06.654877 ip-10-0-130-96 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 18:50:09.772655 ip-10-0-130-96 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:50:09.772655 ip-10-0-130-96 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 18:50:09.772655 ip-10-0-130-96 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:50:09.772655 ip-10-0-130-96 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 18:50:09.772655 ip-10-0-130-96 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:50:09.773409 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.773286    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 18:50:09.777712 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777697    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:09.777712 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777711    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777718    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777722    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777725    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777728    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777731    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777734    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777737    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777740    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777743    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777746    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777749    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777752    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777754    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777757    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777759    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777762    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777765    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777767    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:09.777778 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777770    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777772    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777775    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777778    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777781    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777784    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777786    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777789    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777792    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777795    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777797    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777800    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777802    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777805    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777807    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777810    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777812    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777814    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777817    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777819    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:09.778215 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777822    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777825    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777827    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777830    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777833    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777836    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777839    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777841    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777844    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777846    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777849    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777851    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777854    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777856    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777859    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777862    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777864    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777867    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777869    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777872    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:09.778741 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777874    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777877    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777880    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777882    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777884    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777888    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777891    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777893    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777896    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777898    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777901    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777903    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777906    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777909    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777911    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777914    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777917    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777920    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777923    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777926    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:09.779217 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777929    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777931    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777935    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777939    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777943    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777946    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777949    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777953    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777956    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777958    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777961    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777964    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777967    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777970    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777972    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777975    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777978    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777980    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.777983    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:09.779693 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778730    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778737    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778741    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778744    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778747    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778750    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778753    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778756    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778759    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778763    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778767    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778773    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778777    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778780    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778783    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778785    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778788    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778791    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778794    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778797    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:09.780160 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778800    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778802    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778805    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778815    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778818    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778821    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778823    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778826    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778829    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778831    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778834    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778836    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778839    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778841    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778844    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778847    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778849    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778851    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778854    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778856    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:09.780692 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778859    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778862    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778864    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778867    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778877    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778880    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778883    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778886    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778888    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778891    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778893    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778896    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778899    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778902    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778905    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778907    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778910    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778912    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778915    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778918    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:09.781151 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778920    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778923    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778925    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778928    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778930    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778933    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778935    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778938    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778940    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778943    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778945    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778948    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778951    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778954    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778959    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778962    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778965    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778969    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778971    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:09.781645 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778974    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778976    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778979    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778981    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778984    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778986    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778989    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778991    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778994    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778997    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.778999    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779002    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779004    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779008    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779011    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779014    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779017    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779021    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779023    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779026    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:09.782094 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779406    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779412    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779415    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779418    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779421    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779424    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779427    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779430    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779432    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779435    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779438    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779441    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779443    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779446    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779449    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779451    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779454    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779456    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779459    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:09.782578 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779462    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779464    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779467    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779469    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779472    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779474    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779478    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779480    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779483    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779486    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779488    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779491    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779494    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779496    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779499    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779501    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779504    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779507    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779509    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779512    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:09.783028 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779514    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779517    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779519    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779522    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779539    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779542    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779545    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779548    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779550    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779553    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779555    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779558    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779561    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779563    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779565    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779568    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779571    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779574    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779576    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779579    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:09.783492 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779581    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779587    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779590    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779592    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779595    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779597    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779600    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779603    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779607    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779610    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779613    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779616    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779619    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779622    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779625    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779629    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779632    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779636    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779639    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:09.783976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779642    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779644    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779648    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779651    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779654    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779656    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779659    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779661    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779664    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779667    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779669    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779673    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779676    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779679    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779683    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779687    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779690    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779694    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779697    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779702    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:09.784447 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.779704    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779773    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779780    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779786    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779790    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779794    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779797    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779802    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779806    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779809    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779812    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779816    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779819    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779823    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779825    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779829    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779831    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779834    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779837    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779841    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779844    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779847    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779850    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779853    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 18:50:09.784939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779857    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779860    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779864    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779867    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779870    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779873    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779876    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779881    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779883    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779887    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779890    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779893    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779896    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779899    2569 flags.go:64] FLAG: --enable-server="true"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779902    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779906    2569 flags.go:64] FLAG: --event-burst="100"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779909    2569 flags.go:64] FLAG: --event-qps="50"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779912    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779915    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779917    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779925    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779928    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779932    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779935    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779937    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 18:50:09.785492 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779940    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779943    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779946    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779949    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779951    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779954    2569 flags.go:64] FLAG: --feature-gates=""
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779958    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779961    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779964    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779968    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779971    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779974    2569 flags.go:64] FLAG: --help="false"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779977    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-130-96.ec2.internal"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779980    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779984    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779987    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779990    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779994    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779997    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.779999    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780002    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780005    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780008    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780011    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 18:50:09.786127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780014    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780017    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780019    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780022    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780026    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780029    2569 flags.go:64] FLAG: --lock-file=""
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780031    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780034    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780037    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780042    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780045    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780048    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780051    2569 flags.go:64] FLAG: --logging-format="text"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780053    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780057    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780059    2569 flags.go:64] FLAG: --manifest-url=""
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780062    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780067    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780070    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780074    2569 flags.go:64] FLAG: --max-pods="110"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780077    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780079    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780084    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780087    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780089    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 18:50:09.786786 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780092    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780095    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780103    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780106    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780109    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780113    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780115    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780121    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780124    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780127    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780130    2569 flags.go:64] FLAG: --port="10250"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780133    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780136    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00773548bec4f161e"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780139    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780143    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780145    2569 flags.go:64] FLAG: --register-node="true"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780148    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780151    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780154    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780157    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780160    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780163    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780166    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780169    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780172    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 18:50:09.787480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780176    2569 flags.go:64] FLAG: --runonce="false"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780179    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780182    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780185    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780189    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780191    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780195    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780198    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780201    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780204    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780207    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780210    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780213    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780216    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780218    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780221    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780226    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780229    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780232    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780236    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780240    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780243    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780246    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780249    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780251    2569 flags.go:64] FLAG: --v="2"
Jul 13 18:50:09.788151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780255    2569 flags.go:64] FLAG: --version="false"
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780259    2569 flags.go:64] FLAG: --vmodule=""
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780263    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.780266    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780393    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780397    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780400    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780404    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780407    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780410    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780413    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780416    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780420    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780424    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780427    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780430    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780432    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780435    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780438    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780441    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:09.788837 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780444    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780447    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780450    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780452    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780455    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780458    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780460    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780463    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780466    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780468    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780471    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780474    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780476    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780479    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780482    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780484    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780487    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780489    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780492    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780496    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:09.789383 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780498    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780501    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780503    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780506    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780509    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780511    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780514    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780517    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780519    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780522    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780538    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780542    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780545    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780548    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780550    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780553    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780556    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780560    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780563    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:09.789980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780566    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780569    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780572    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780575    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780578    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780580    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780583    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780586    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780588    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780591    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780593    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780596    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780600    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780603    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780605    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780608    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780611    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780614    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780617    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:09.790477 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780619    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780622    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780625    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780627    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780630    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780633    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780635    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780638    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780641    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780643    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780646    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780648    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780651    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780654    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780657    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780659    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780668    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780671    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780673    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780676    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:09.791033 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780678    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780683    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780685    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780688    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.780691    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.781404    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.787976    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.787992    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.788005    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.788015    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788069    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788074    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788077    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788080    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788084    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:09.791519 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788087    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788090    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788093    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788096    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788099    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788102    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788105    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788108    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788111    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788114    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788117    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788120    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788122    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788125    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788128    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788130    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788133    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788135    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788138    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788141    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:09.791908 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788143    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788146    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788149    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788152    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788160    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788163    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788166    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788169    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788171    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788174    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788176    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788179    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788182    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788185    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788187    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788190    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788193    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788195    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788198    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788201    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:09.792381 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788203    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788206    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788209    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788211    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788214    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788217    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788219    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788222    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788225    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788229    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788233    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788236    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788239    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788241    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788244    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788247    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788249    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788253    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788257    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:09.792863 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788261    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788264    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788267    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788271    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788274    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788277    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788279    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788282    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788285    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788288    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788290    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788293    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788295    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788298    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788300    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788303    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788305    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788308    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788310    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:09.793321 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788313    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788316    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788319    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788321    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788324    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788327    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788329    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788332    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788335    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788337    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788340    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788342    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788345    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788349    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788351    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:09.793792 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788354    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.788359    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788467    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788472    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788475    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788478    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788481    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788484    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788487    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788490    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788493    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788496    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788499    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788502    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788505    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:09.794144 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788508    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788510    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788513    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788516    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788518    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788521    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788523    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788539    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788542    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788545    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788547    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788550    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788553    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788555    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788558    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788561    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788564    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788566    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788569    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788572    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:09.794494 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788575    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788577    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788580    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788583    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788585    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788588    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788591    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788593    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788596    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788599    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788602    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788605    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788608    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788610    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788613    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788616    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788619    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788623    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788627    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:09.794976 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788630    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788633    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788636    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788638    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788641    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788644    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788646    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788649    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788651    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788655    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788657    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788660    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788662    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788666    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788670    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788673    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788675    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788678    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788680    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788683    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:09.795456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788685    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788688    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788691    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788693    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788696    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788699    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788701    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788704    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788707    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788709    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788712    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788715    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788717    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788720    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788722    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788725    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788728    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788730    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788733    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:09.795965 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788737    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788740    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788743    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788747    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788750    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788753    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788756    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:09.788758    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.788764    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.788868    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.790749    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.791951    2569 server.go:1019] "Starting client certificate rotation"
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.792060    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:50:09.796400 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.792091    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:50:09.796949 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.794000    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:50:09.796949 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.796836    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:50:09.806346 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.806323    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 13 18:50:09.808509 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.808495    2569 log.go:25] "Validated CRI v1 image API"
Jul 13 18:50:09.809363 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.809337    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 18:50:09.813187 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.813169    2569 fs.go:135] Filesystem UUIDs: map[0de3c233-8d02-4e8f-bf33-b774d3c0b939:/dev/nvme0n1p4 64e1a32e-8e0d-4f3f-9c14-c4bbdb870128:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 13 18:50:09.813249 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.813187    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 18:50:09.818839 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.818716    2569 manager.go:216] Machine: {Timestamp:2026-07-13 18:50:09.817650833 +0000 UTC m=+1.923254443 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3124224 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d73b65f456eeafeadfbf9073a5dac SystemUUID:ec2d73b6-5f45-6eea-fead-fbf9073a5dac BootID:82fc1b98-a32f-44df-ad77-893b1cd99aac Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d4:91:61:2d:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d4:91:61:2d:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:a6:6f:b8:09:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 18:50:09.818839 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.818833    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 18:50:09.818976 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.818908    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 18:50:09.819567 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.819551    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:50:09.821957 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.821929    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 18:50:09.822112 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.821963    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 18:50:09.822601 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.822590    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 18:50:09.822652 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.822606    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 18:50:09.822652 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.822616    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:50:09.824004 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.823992    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:50:09.824784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.824773    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 18:50:09.827471 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.827458    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:50:09.827618 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.827607    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 18:50:09.827674 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.827648    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 18:50:09.827674 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.827662    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 18:50:09.827772 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.827682    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 18:50:09.827772 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.827695    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 13 18:50:09.827772 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.827713    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 18:50:09.828116 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.828103    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:50:09.828184 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.828126    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:50:09.830049 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.830032    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 18:50:09.831370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.831356    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 18:50:09.831422 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.831386    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 18:50:09.831953 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.831941    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 18:50:09.831999 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.831958    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 18:50:09.831999 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.831964    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 18:50:09.831999 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.831970    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 18:50:09.831999 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.831978    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 18:50:09.831999 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.831984    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 18:50:09.832139 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.832029    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 18:50:09.832139 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.832035    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 18:50:09.832139 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.832051    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 18:50:09.832139 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.832057    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 18:50:09.832139 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.832071    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 18:50:09.832139 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.832079    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 18:50:09.832935 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.832924    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 18:50:09.832935 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.832934    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 18:50:09.835356 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.835336    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 18:50:09.835462 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.835442    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 18:50:09.835511 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.835445    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 18:50:09.836005 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.835993    2569 server.go:1268] "Started kubelet"
Jul 13 18:50:09.837362 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.837331    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 18:50:09.838007 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.837951    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 18:50:09.838109 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.838031    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 18:50:09.838649 ip-10-0-130-96 systemd[1]: Started Kubernetes Kubelet.
Jul 13 18:50:09.840294 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.840173    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 18:50:09.840294 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.840286    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 18:50:09.844096 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.844080    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 18:50:09.844096 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.844080    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 18:50:09.844275 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.844255    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:09.844734 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.844622    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 18:50:09.844734 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.844637    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 18:50:09.844734 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.844653    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 18:50:09.845432 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.845416    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 18:50:09.845432 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.845428    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 18:50:09.845628 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.845516    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 18:50:09.845890 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.845866    2569 factory.go:153] Registering CRI-O factory
Jul 13 18:50:09.845968 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.845915    2569 factory.go:223] Registration of the crio container factory successfully
Jul 13 18:50:09.845968 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.845966    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 18:50:09.846062 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.845975    2569 factory.go:55] Registering systemd factory
Jul 13 18:50:09.846062 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.845982    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 13 18:50:09.846062 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.846002    2569 factory.go:103] Registering Raw factory
Jul 13 18:50:09.846062 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.846014    2569 manager.go:1196] Started watching for new ooms in manager
Jul 13 18:50:09.846397 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.846384    2569 manager.go:349] Starting recovery of all containers
Jul 13 18:50:09.846750 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.846728    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 18:50:09.847698 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.846895    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-96.ec2.internal.18c1edffafb8176e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-96.ec2.internal,UID:ip-10-0-130-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-96.ec2.internal,},FirstTimestamp:2026-07-13 18:50:09.835947886 +0000 UTC m=+1.941551495,LastTimestamp:2026-07-13 18:50:09.835947886 +0000 UTC m=+1.941551495,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-96.ec2.internal,}"
Jul 13 18:50:09.851362 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.851340    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bpxns"
Jul 13 18:50:09.854131 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.854071    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 18:50:09.854972 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.854954    2569 manager.go:354] Recovery completed
Jul 13 18:50:09.857692 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.857667    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 18:50:09.857800 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.857783    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bpxns"
Jul 13 18:50:09.860417 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.860406    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:09.862257 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.862242    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:09.862324 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.862284    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:09.862324 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.862296    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:09.862994 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.862980    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 18:50:09.862994 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.862995    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 18:50:09.863123 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.863012    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:50:09.863123 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.862982    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:09.863716 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.863702    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:09.863773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.863727    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:09.863773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.863736    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:09.864053 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.864041    2569 policy_none.go:49] "None policy: Start"
Jul 13 18:50:09.864104 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.864061    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 18:50:09.864104 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.864074    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 18:50:09.865114 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.865101    2569 policy_none.go:47] "Start"
Jul 13 18:50:09.900275 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.900100    2569 manager.go:337] "Starting Device Plugin manager"
Jul 13 18:50:09.900275 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.900264    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 18:50:09.900402 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.900277    2569 server.go:87] "Starting device plugin registration server"
Jul 13 18:50:09.900476 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.900465    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 18:50:09.900606 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.900477    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 18:50:09.900606 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.900590    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 18:50:09.900754 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.900737    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 18:50:09.900754 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:09.900752    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 18:50:09.901137 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.901118    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 18:50:09.901214 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:09.901171    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.001993 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.001966    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:10.002984 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.002892    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:10.002984 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.002922    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:10.002984 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.002936    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:10.002984 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.002962    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.009013 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.008994    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.009101 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.009017    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-96.ec2.internal\": node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.027269 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.027249    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.039892 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.039866    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 18:50:10.039973 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.039908    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 18:50:10.039973 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.039931    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 18:50:10.039973 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.039966    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 18:50:10.042072 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.042051    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:10.127525 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.127482    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.140671 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.140640    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal"]
Jul 13 18:50:10.140739 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.140722    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:10.141817 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.141796    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:10.141928 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.141833    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:10.141928 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.141849    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:10.143296 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.143283    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:10.143424 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.143407    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.143467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.143439    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:10.144037 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.144024    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:10.144110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.144037    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:10.144110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.144053    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:10.144110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.144057    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:10.144110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.144067    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:10.144110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.144079    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:10.145190 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.145177    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.145250 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.145202    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:10.145860 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.145837    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:10.145926 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.145865    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:10.145926 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.145877    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:10.145994 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.145926    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/908b719ae1c9201ec6cd9981720d3cfe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal\" (UID: \"908b719ae1c9201ec6cd9981720d3cfe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.145994 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.145952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908b719ae1c9201ec6cd9981720d3cfe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal\" (UID: \"908b719ae1c9201ec6cd9981720d3cfe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.145994 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.145968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfc6c26c84bf408e478864f003148e2a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-96.ec2.internal\" (UID: \"cfc6c26c84bf408e478864f003148e2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.169325 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.169262    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-96.ec2.internal\" not found" node="ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.173710 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.173691    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-96.ec2.internal\" not found" node="ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.227641 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.227607    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.246653 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.246621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/908b719ae1c9201ec6cd9981720d3cfe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal\" (UID: \"908b719ae1c9201ec6cd9981720d3cfe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.246799 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.246656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908b719ae1c9201ec6cd9981720d3cfe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal\" (UID: \"908b719ae1c9201ec6cd9981720d3cfe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.246799 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.246677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfc6c26c84bf408e478864f003148e2a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-96.ec2.internal\" (UID: \"cfc6c26c84bf408e478864f003148e2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.246799 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.246730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfc6c26c84bf408e478864f003148e2a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-96.ec2.internal\" (UID: \"cfc6c26c84bf408e478864f003148e2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.246799 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.246731    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/908b719ae1c9201ec6cd9981720d3cfe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal\" (UID: \"908b719ae1c9201ec6cd9981720d3cfe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.246799 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.246745    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908b719ae1c9201ec6cd9981720d3cfe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal\" (UID: \"908b719ae1c9201ec6cd9981720d3cfe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.327941 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.327903    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.428557 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.428463    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.472014 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.471990    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.476914 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.476893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:10.529604 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.529581    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.630084 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.630050    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.730583 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.730492    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.746489 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.746465    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:10.792100 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.792078    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 18:50:10.792489 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.792191    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:50:10.792489 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.792223    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:50:10.831518 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.831493    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:10.861247 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.861207    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 18:45:09 +0000 UTC" deadline="2028-02-24 13:53:12.773784647 +0000 UTC"
Jul 13 18:50:10.861339 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:10.861250    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14179h3m1.912537319s"
Jul 13 18:50:10.932595 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:10.932566    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:11.006984 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:11.006949    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908b719ae1c9201ec6cd9981720d3cfe.slice/crio-74f9bb066e3c4802f68ad085b16b735a5a8540c444ecaa016ff46e458f30d6d6 WatchSource:0}: Error finding container 74f9bb066e3c4802f68ad085b16b735a5a8540c444ecaa016ff46e458f30d6d6: Status 404 returned error can't find the container with id 74f9bb066e3c4802f68ad085b16b735a5a8540c444ecaa016ff46e458f30d6d6
Jul 13 18:50:11.010877 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.010861    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:50:11.033554 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:11.033517    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:11.042546 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.042489    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal" event={"ID":"908b719ae1c9201ec6cd9981720d3cfe","Type":"ContainerStarted","Data":"74f9bb066e3c4802f68ad085b16b735a5a8540c444ecaa016ff46e458f30d6d6"}
Jul 13 18:50:11.134053 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:11.134014    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-96.ec2.internal\" not found"
Jul 13 18:50:11.191240 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.191216    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:11.234060 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.234039    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:11.245390 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.245370    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:11.255682 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.255619    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:50:11.256613 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.256601    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal"
Jul 13 18:50:11.267132 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.267116    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:50:11.438456 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:11.438428    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc6c26c84bf408e478864f003148e2a.slice/crio-92f9aa9a29d5f030ad194f49d804e8386d48e3b44fe12451b5412816abb3f3dd WatchSource:0}: Error finding container 92f9aa9a29d5f030ad194f49d804e8386d48e3b44fe12451b5412816abb3f3dd: Status 404 returned error can't find the container with id 92f9aa9a29d5f030ad194f49d804e8386d48e3b44fe12451b5412816abb3f3dd
Jul 13 18:50:11.828460 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.828428    2569 apiserver.go:52] "Watching apiserver"
Jul 13 18:50:11.836314 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.836284    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 18:50:11.837281 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.837252    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-5kc4j","openshift-network-operator/iptables-alerter-mdpdw","openshift-ovn-kubernetes/ovnkube-node-wftcq","kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal","openshift-cluster-node-tuning-operator/tuned-599db","openshift-dns/node-resolver-z7v8z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal","openshift-multus/multus-864b4","openshift-multus/multus-additional-cni-plugins-nkdxv","kube-system/global-pull-secret-syncer-pg7df","kube-system/konnectivity-agent-2l425","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8","openshift-image-registry/node-ca-bpjbm","openshift-multus/network-metrics-daemon-pdlbh"]
Jul 13 18:50:11.838839 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.838791    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.839863 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.839844    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.841168 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.841148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.842612 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.842593    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:11.842745 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:11.842692    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:11.844296 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.844152    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:11.844296 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.844170    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:11.844296 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.844259    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.844457 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.844398    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 18:50:11.845193 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.844677    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 18:50:11.845285 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.845219    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 18:50:11.845347 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.845303    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 18:50:11.845398 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.845349    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cpv6t\""
Jul 13 18:50:11.845398 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.845351    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 18:50:11.845564 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.845516    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q5788\""
Jul 13 18:50:11.845638 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.845601    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.847297 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.846820    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:11.847297 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:11.846883    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:11.848265 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.848244    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.848983 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.848957    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 18:50:11.849324 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.849302    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 18:50:11.849487 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.849470    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 18:50:11.850738 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.850713    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:11.851103 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.851088    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 18:50:11.851176 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.851102    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tjkkw\""
Jul 13 18:50:11.851298 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.851279    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 18:50:11.852022 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.852002    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 18:50:11.852338 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.852320    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:11.852569 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.852554    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hmncx\""
Jul 13 18:50:11.854107 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.852627    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:11.854843 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.854726    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:11.854843 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:11.854832    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:11.855391 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.855363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1927b946-784d-4d94-b311-3eb5824fa9ce-host-slash\") pod \"iptables-alerter-mdpdw\" (UID: \"1927b946-784d-4d94-b311-3eb5824fa9ce\") " pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.855875 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.855853    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-env-overrides\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.855960 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.855890    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-kubernetes\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.855960 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.855940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-sysctl-conf\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.856071 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.855963    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-sys\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.856071 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.855986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/103a12f9-a611-452d-ab26-317a59123f8b-etc-tuned\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.856071 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856023    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdpgx\" (UniqueName: \"kubernetes.io/projected/7f27934f-42fa-4166-a6ef-eb8062375c7e-kube-api-access-sdpgx\") pod \"node-resolver-z7v8z\" (UID: \"7f27934f-42fa-4166-a6ef-eb8062375c7e\") " pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.856071 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856048    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52a0b632-455c-4340-99f2-d4178918e76c-host\") pod \"node-ca-bpjbm\" (UID: \"52a0b632-455c-4340-99f2-d4178918e76c\") " pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.856264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856070    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52a0b632-455c-4340-99f2-d4178918e76c-serviceca\") pod \"node-ca-bpjbm\" (UID: \"52a0b632-455c-4340-99f2-d4178918e76c\") " pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.856264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856091    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 18:50:11.856264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856130    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1927b946-784d-4d94-b311-3eb5824fa9ce-iptables-alerter-script\") pod \"iptables-alerter-mdpdw\" (UID: \"1927b946-784d-4d94-b311-3eb5824fa9ce\") " pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.856264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856179    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-ovn-node-metrics-cert\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856203    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5xc\" (UniqueName: \"kubernetes.io/projected/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-kube-api-access-tr5xc\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-modprobe-d\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.856264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856250    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 18:50:11.856264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856260    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzngw\" (UniqueName: \"kubernetes.io/projected/52a0b632-455c-4340-99f2-d4178918e76c-kube-api-access-jzngw\") pod \"node-ca-bpjbm\" (UID: \"52a0b632-455c-4340-99f2-d4178918e76c\") " pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92wb9\" (UniqueName: \"kubernetes.io/projected/1927b946-784d-4d94-b311-3eb5824fa9ce-kube-api-access-92wb9\") pod \"iptables-alerter-mdpdw\" (UID: \"1927b946-784d-4d94-b311-3eb5824fa9ce\") " pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856305    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-kubelet\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-run-openvswitch\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856352    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-cni-netd\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-host\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856453    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-systemd-units\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-slash\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-run-ovn\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856616    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-ovnkube-config\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.856660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856640    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/103a12f9-a611-452d-ab26-317a59123f8b-tmp\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfvs\" (UniqueName: \"kubernetes.io/projected/103a12f9-a611-452d-ab26-317a59123f8b-kube-api-access-tzfvs\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856702    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f27934f-42fa-4166-a6ef-eb8062375c7e-hosts-file\") pod \"node-resolver-z7v8z\" (UID: \"7f27934f-42fa-4166-a6ef-eb8062375c7e\") " pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856713    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q5z5w\""
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-run-netns\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-node-log\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-ovnkube-script-lib\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-sysctl-d\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-var-lib-openvswitch\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856860    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-var-lib-kubelet\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856894    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f27934f-42fa-4166-a6ef-eb8062375c7e-tmp-dir\") pod \"node-resolver-z7v8z\" (UID: \"7f27934f-42fa-4166-a6ef-eb8062375c7e\") " pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856918    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-run-systemd\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-etc-openvswitch\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856953    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856964    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-cni-bin\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.856985    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-log-socket\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-sysconfig\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.857217 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857044    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 18:50:11.858075 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-systemd\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.858075 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857077    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-run\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.858075 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857100    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-lib-modules\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.858075 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857175    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vls74\""
Jul 13 18:50:11.858075 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857300    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:11.858075 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857404    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 18:50:11.858075 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.857576    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 18:50:11.858951 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.858930    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:11.860066 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.860050    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 18:50:11.868919 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.868767    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 18:50:11.868919 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.868801    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9pg8w\""
Jul 13 18:50:11.869062 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.869051    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 18:50:11.869244 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.869219    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k5vbp\""
Jul 13 18:50:11.869356 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.869338    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 18:50:11.870298 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.870177    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 18:50:11.871376 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.871355    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 18:50:11.871376 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.871369    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 18:50:11.871937 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.871920    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wc5nl\""
Jul 13 18:50:11.888693 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.888675    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:50:11.918792 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.918767    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-srzph"
Jul 13 18:50:11.933031 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.933006    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-srzph"
Jul 13 18:50:11.945676 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.945657    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 18:50:11.957456 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-var-lib-openvswitch\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.957587 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-var-lib-kubelet\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.957587 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957523    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-cni-dir\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.957587 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-var-lib-openvswitch\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.957587 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-socket-dir-parent\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9123bd54-a028-460c-961b-82b16be6168f-cni-binary-copy\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9123bd54-a028-460c-961b-82b16be6168f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7ee3d-6283-41fc-822a-0597c41cdccd-konnectivity-ca\") pod \"konnectivity-agent-2l425\" (UID: \"7ef7ee3d-6283-41fc-822a-0597c41cdccd\") " pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957631    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-var-lib-kubelet\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957647    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-sys-fs\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-run-systemd\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-etc-openvswitch\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957715    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-run-systemd\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957750    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:11.957784 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957754    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-etc-openvswitch\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9123bd54-a028-460c-961b-82b16be6168f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957823    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957847    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-sysconfig\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkb2\" (UniqueName: \"kubernetes.io/projected/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-kube-api-access-rzkb2\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957911    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-env-overrides\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.957978    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-sysconfig\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-kubernetes\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-cnibin\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958159    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-kubernetes\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.958233 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958219    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-registration-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52a0b632-455c-4340-99f2-d4178918e76c-host\") pod \"node-ca-bpjbm\" (UID: \"52a0b632-455c-4340-99f2-d4178918e76c\") " pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52a0b632-455c-4340-99f2-d4178918e76c-serviceca\") pod \"node-ca-bpjbm\" (UID: \"52a0b632-455c-4340-99f2-d4178918e76c\") " pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1927b946-784d-4d94-b311-3eb5824fa9ce-iptables-alerter-script\") pod \"iptables-alerter-mdpdw\" (UID: \"1927b946-784d-4d94-b311-3eb5824fa9ce\") " pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958382    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-cni-binary-copy\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958325    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52a0b632-455c-4340-99f2-d4178918e76c-host\") pod \"node-ca-bpjbm\" (UID: \"52a0b632-455c-4340-99f2-d4178918e76c\") " pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958405    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-run-netns\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958460    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-run-multus-certs\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958501    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzngw\" (UniqueName: \"kubernetes.io/projected/52a0b632-455c-4340-99f2-d4178918e76c-kube-api-access-jzngw\") pod \"node-ca-bpjbm\" (UID: \"52a0b632-455c-4340-99f2-d4178918e76c\") " pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958547    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92wb9\" (UniqueName: \"kubernetes.io/projected/1927b946-784d-4d94-b311-3eb5824fa9ce-kube-api-access-92wb9\") pod \"iptables-alerter-mdpdw\" (UID: \"1927b946-784d-4d94-b311-3eb5824fa9ce\") " pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958548    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-env-overrides\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-run-openvswitch\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-host\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958634    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfvs\" (UniqueName: \"kubernetes.io/projected/103a12f9-a611-452d-ab26-317a59123f8b-kube-api-access-tzfvs\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f27934f-42fa-4166-a6ef-eb8062375c7e-hosts-file\") pod \"node-resolver-z7v8z\" (UID: \"7f27934f-42fa-4166-a6ef-eb8062375c7e\") " pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958654    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-run-openvswitch\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958682    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-host\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.958708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958690    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-var-lib-cni-multus\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958722    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-slash\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-var-lib-kubelet\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958759    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f27934f-42fa-4166-a6ef-eb8062375c7e-hosts-file\") pod \"node-resolver-z7v8z\" (UID: \"7f27934f-42fa-4166-a6ef-eb8062375c7e\") " pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-system-cni-dir\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958819    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-run-netns\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52a0b632-455c-4340-99f2-d4178918e76c-serviceca\") pod \"node-ca-bpjbm\" (UID: \"52a0b632-455c-4340-99f2-d4178918e76c\") " pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958845    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-ovnkube-script-lib\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-slash\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958869    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958877    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-run-netns\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958891    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-os-release\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03610f49-0413-408e-9e5c-d1adc6d46204-kubelet-config\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958946    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2tz5\" (UniqueName: \"kubernetes.io/projected/9301dc45-e443-459d-a18f-026df166a802-kube-api-access-b2tz5\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-run-k8s-cni-cncf-io\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.958991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-etc-kubernetes\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03610f49-0413-408e-9e5c-d1adc6d46204-dbus\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:11.959467 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959035    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-device-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-cni-bin\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959091    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959113    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-cni-bin\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-log-socket\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-systemd\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959240    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-log-socket\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959336    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-ovnkube-script-lib\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959864    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-run\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-lib-modules\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959964    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-systemd\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.959998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-system-cni-dir\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-hostroot\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1927b946-784d-4d94-b311-3eb5824fa9ce-host-slash\") pod \"iptables-alerter-mdpdw\" (UID: \"1927b946-784d-4d94-b311-3eb5824fa9ce\") " pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960112    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-run\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-lib-modules\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.960205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-sysctl-conf\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960157    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1927b946-784d-4d94-b311-3eb5824fa9ce-host-slash\") pod \"iptables-alerter-mdpdw\" (UID: \"1927b946-784d-4d94-b311-3eb5824fa9ce\") " pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-sys\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960245    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-sysctl-conf\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960248    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/103a12f9-a611-452d-ab26-317a59123f8b-etc-tuned\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960275    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-sys\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960303    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdpgx\" (UniqueName: \"kubernetes.io/projected/7f27934f-42fa-4166-a6ef-eb8062375c7e-kube-api-access-sdpgx\") pod \"node-resolver-z7v8z\" (UID: \"7f27934f-42fa-4166-a6ef-eb8062375c7e\") " pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-ovn-node-metrics-cert\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5xc\" (UniqueName: \"kubernetes.io/projected/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-kube-api-access-tr5xc\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-modprobe-d\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mbh\" (UniqueName: \"kubernetes.io/projected/9123bd54-a028-460c-961b-82b16be6168f-kube-api-access-g4mbh\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960455    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-socket-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f27934f-42fa-4166-a6ef-eb8062375c7e-tmp-dir\") pod \"node-resolver-z7v8z\" (UID: \"7f27934f-42fa-4166-a6ef-eb8062375c7e\") " pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960569    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-cnibin\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-kubelet\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-cni-netd\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960662    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-kubelet\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960662    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:11.961009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960688    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960694    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1927b946-784d-4d94-b311-3eb5824fa9ce-iptables-alerter-script\") pod \"iptables-alerter-mdpdw\" (UID: \"1927b946-784d-4d94-b311-3eb5824fa9ce\") " pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960709    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-daemon-config\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960734    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-systemd-units\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960762    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-run-ovn\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-modprobe-d\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960711    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-cni-netd\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960792    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-systemd-units\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960877    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f27934f-42fa-4166-a6ef-eb8062375c7e-tmp-dir\") pod \"node-resolver-z7v8z\" (UID: \"7f27934f-42fa-4166-a6ef-eb8062375c7e\") " pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960909    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-ovnkube-config\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960940    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960938    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-run-ovn\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.960980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/103a12f9-a611-452d-ab26-317a59123f8b-tmp\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.961084    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5w92\" (UniqueName: \"kubernetes.io/projected/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-kube-api-access-w5w92\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.961159    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-conf-dir\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.961211    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-os-release\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:11.961766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.961259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-node-log\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.962436 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.961899    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-node-log\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.962436 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.962114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-sysctl-d\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.962436 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.962162    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-var-lib-cni-bin\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:11.962436 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.962195    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ef7ee3d-6283-41fc-822a-0597c41cdccd-agent-certs\") pod \"konnectivity-agent-2l425\" (UID: \"7ef7ee3d-6283-41fc-822a-0597c41cdccd\") " pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:11.962436 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.962255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-etc-selinux\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:11.962436 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.962270    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-ovnkube-config\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.962719 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.962435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/103a12f9-a611-452d-ab26-317a59123f8b-etc-sysctl-d\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.964782 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.964756    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/103a12f9-a611-452d-ab26-317a59123f8b-etc-tuned\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.966321 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.966299    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/103a12f9-a611-452d-ab26-317a59123f8b-tmp\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.966416 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.966356    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-ovn-node-metrics-cert\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:11.967562 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:11.967246    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:11.967562 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:11.967269    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:11.967562 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:11.967283    2569 projected.go:196] Error preparing data for projected volume kube-api-access-dzvfr for pod openshift-network-diagnostics/network-check-target-5kc4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:11.967562 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:11.967361    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr podName:8d506c0b-8324-498e-962e-6e396a66159e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:12.467326471 +0000 UTC m=+4.572930088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dzvfr" (UniqueName: "kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr") pod "network-check-target-5kc4j" (UID: "8d506c0b-8324-498e-962e-6e396a66159e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:11.967963 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.967941    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfvs\" (UniqueName: \"kubernetes.io/projected/103a12f9-a611-452d-ab26-317a59123f8b-kube-api-access-tzfvs\") pod \"tuned-599db\" (UID: \"103a12f9-a611-452d-ab26-317a59123f8b\") " pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:11.968515 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.968471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92wb9\" (UniqueName: \"kubernetes.io/projected/1927b946-784d-4d94-b311-3eb5824fa9ce-kube-api-access-92wb9\") pod \"iptables-alerter-mdpdw\" (UID: \"1927b946-784d-4d94-b311-3eb5824fa9ce\") " pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:11.968704 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.968683    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzngw\" (UniqueName: \"kubernetes.io/projected/52a0b632-455c-4340-99f2-d4178918e76c-kube-api-access-jzngw\") pod \"node-ca-bpjbm\" (UID: \"52a0b632-455c-4340-99f2-d4178918e76c\") " pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:11.970066 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.970041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdpgx\" (UniqueName: \"kubernetes.io/projected/7f27934f-42fa-4166-a6ef-eb8062375c7e-kube-api-access-sdpgx\") pod \"node-resolver-z7v8z\" (UID: \"7f27934f-42fa-4166-a6ef-eb8062375c7e\") " pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:11.971012 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:11.970992    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5xc\" (UniqueName: \"kubernetes.io/projected/7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7-kube-api-access-tr5xc\") pod \"ovnkube-node-wftcq\" (UID: \"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:12.044579 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.044544    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal" event={"ID":"cfc6c26c84bf408e478864f003148e2a","Type":"ContainerStarted","Data":"92f9aa9a29d5f030ad194f49d804e8386d48e3b44fe12451b5412816abb3f3dd"}
Jul 13 18:50:12.063351 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063318    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.063509 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063364    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-system-cni-dir\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063509 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-hostroot\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063509 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mbh\" (UniqueName: \"kubernetes.io/projected/9123bd54-a028-460c-961b-82b16be6168f-kube-api-access-g4mbh\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.063509 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-socket-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.063509 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-cnibin\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063509 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-system-cni-dir\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063551    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-hostroot\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-cnibin\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063574    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063584    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-daemon-config\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063630    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5w92\" (UniqueName: \"kubernetes.io/projected/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-kube-api-access-w5w92\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-conf-dir\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-os-release\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063714    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-var-lib-cni-bin\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063751    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-conf-dir\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063756    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-var-lib-cni-bin\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063633    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-socket-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.063816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063795    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-os-release\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063834    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ef7ee3d-6283-41fc-822a-0597c41cdccd-agent-certs\") pod \"konnectivity-agent-2l425\" (UID: \"7ef7ee3d-6283-41fc-822a-0597c41cdccd\") " pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063871    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-etc-selinux\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063897    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-cni-dir\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063929    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-socket-dir-parent\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9123bd54-a028-460c-961b-82b16be6168f-cni-binary-copy\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.063980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9123bd54-a028-460c-961b-82b16be6168f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064007    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7ee3d-6283-41fc-822a-0597c41cdccd-konnectivity-ca\") pod \"konnectivity-agent-2l425\" (UID: \"7ef7ee3d-6283-41fc-822a-0597c41cdccd\") " pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064031    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-socket-dir-parent\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-sys-fs\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064113    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9123bd54-a028-460c-961b-82b16be6168f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-daemon-config\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064202    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkb2\" (UniqueName: \"kubernetes.io/projected/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-kube-api-access-rzkb2\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064231    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-cnibin\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064271    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-registration-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.064370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064299    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-cni-binary-copy\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064327    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-run-netns\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-run-multus-certs\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064385    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-var-lib-cni-multus\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-var-lib-kubelet\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-system-cni-dir\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064507    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-os-release\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03610f49-0413-408e-9e5c-d1adc6d46204-kubelet-config\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2tz5\" (UniqueName: \"kubernetes.io/projected/9301dc45-e443-459d-a18f-026df166a802-kube-api-access-b2tz5\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064615    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9123bd54-a028-460c-961b-82b16be6168f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064606    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9123bd54-a028-460c-961b-82b16be6168f-cni-binary-copy\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-run-k8s-cni-cncf-io\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064650    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7ee3d-6283-41fc-822a-0597c41cdccd-konnectivity-ca\") pod \"konnectivity-agent-2l425\" (UID: \"7ef7ee3d-6283-41fc-822a-0597c41cdccd\") " pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064650    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-multus-cni-dir\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-etc-kubernetes\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064684    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-run-k8s-cni-cncf-io\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065173 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064704    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03610f49-0413-408e-9e5c-d1adc6d46204-dbus\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064704    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-etc-kubernetes\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-var-lib-cni-multus\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064744    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-etc-selinux\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.064751    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064756    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03610f49-0413-408e-9e5c-d1adc6d46204-dbus\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-device-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064748    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-sys-fs\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064776    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-var-lib-kubelet\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.064811    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret podName:03610f49-0413-408e-9e5c-d1adc6d46204 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:12.564793198 +0000 UTC m=+4.670396833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret") pod "global-pull-secret-syncer-pg7df" (UID: "03610f49-0413-408e-9e5c-d1adc6d46204") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03610f49-0413-408e-9e5c-d1adc6d46204-kubelet-config\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.064842    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064681    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-run-netns\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064850    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-device-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064887    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-registration-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064899    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9301dc45-e443-459d-a18f-026df166a802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.064911    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs podName:c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:12.564894749 +0000 UTC m=+4.670498352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs") pod "network-metrics-daemon-pdlbh" (UID: "c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:12.065911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064810    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-os-release\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.066657 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064948    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-host-run-multus-certs\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.066657 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.064986    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-system-cni-dir\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.066657 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.065005    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9123bd54-a028-460c-961b-82b16be6168f-cnibin\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.066657 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.065341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-cni-binary-copy\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.066657 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.065341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9123bd54-a028-460c-961b-82b16be6168f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.066657 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.066581    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ef7ee3d-6283-41fc-822a-0597c41cdccd-agent-certs\") pod \"konnectivity-agent-2l425\" (UID: \"7ef7ee3d-6283-41fc-822a-0597c41cdccd\") " pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:12.073841 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.073806    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2tz5\" (UniqueName: \"kubernetes.io/projected/9301dc45-e443-459d-a18f-026df166a802-kube-api-access-b2tz5\") pod \"aws-ebs-csi-driver-node-qs9b8\" (UID: \"9301dc45-e443-459d-a18f-026df166a802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.074115 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.074072    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mbh\" (UniqueName: \"kubernetes.io/projected/9123bd54-a028-460c-961b-82b16be6168f-kube-api-access-g4mbh\") pod \"multus-additional-cni-plugins-nkdxv\" (UID: \"9123bd54-a028-460c-961b-82b16be6168f\") " pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.075283 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.075266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5w92\" (UniqueName: \"kubernetes.io/projected/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-kube-api-access-w5w92\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:12.075794 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.075768    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkb2\" (UniqueName: \"kubernetes.io/projected/bd43cbf1-95fb-4033-8a3a-bc8d523c80ae-kube-api-access-rzkb2\") pod \"multus-864b4\" (UID: \"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae\") " pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.088555 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.088493    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:12.151239 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.151208    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bpjbm"
Jul 13 18:50:12.162223 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.162190    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mdpdw"
Jul 13 18:50:12.169086 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.169063    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:12.173865 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.173845    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-599db"
Jul 13 18:50:12.179767 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.179739    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z7v8z"
Jul 13 18:50:12.187451 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.187427    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-864b4"
Jul 13 18:50:12.194426 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.194407    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nkdxv"
Jul 13 18:50:12.201182 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.201163    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:12.205840 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.205821    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8"
Jul 13 18:50:12.207695 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:12.207671    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a0b632_455c_4340_99f2_d4178918e76c.slice/crio-872883d56085b75af88869cc1e3c8d4f4423e9336566c172ca50b5e3ac809897 WatchSource:0}: Error finding container 872883d56085b75af88869cc1e3c8d4f4423e9336566c172ca50b5e3ac809897: Status 404 returned error can't find the container with id 872883d56085b75af88869cc1e3c8d4f4423e9336566c172ca50b5e3ac809897
Jul 13 18:50:12.214261 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:12.214237    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1927b946_784d_4d94_b311_3eb5824fa9ce.slice/crio-7ede58d3b2500c4b4b6ab511cfa981722e5e4d196610a21b4757e17c53f6a0a5 WatchSource:0}: Error finding container 7ede58d3b2500c4b4b6ab511cfa981722e5e4d196610a21b4757e17c53f6a0a5: Status 404 returned error can't find the container with id 7ede58d3b2500c4b4b6ab511cfa981722e5e4d196610a21b4757e17c53f6a0a5
Jul 13 18:50:12.214926 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:12.214707    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f27934f_42fa_4166_a6ef_eb8062375c7e.slice/crio-1001a436612d90ae537bac2ed226257f21aab29f8a9ec73290f471e9dbc566f4 WatchSource:0}: Error finding container 1001a436612d90ae537bac2ed226257f21aab29f8a9ec73290f471e9dbc566f4: Status 404 returned error can't find the container with id 1001a436612d90ae537bac2ed226257f21aab29f8a9ec73290f471e9dbc566f4
Jul 13 18:50:12.216919 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:12.216748    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103a12f9_a611_452d_ab26_317a59123f8b.slice/crio-6a501fecf2a607485fc917988e01f1d5370c9d9cb94238f14fcd905ffee7a945 WatchSource:0}: Error finding container 6a501fecf2a607485fc917988e01f1d5370c9d9cb94238f14fcd905ffee7a945: Status 404 returned error can't find the container with id 6a501fecf2a607485fc917988e01f1d5370c9d9cb94238f14fcd905ffee7a945
Jul 13 18:50:12.221320 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:12.221297    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd43cbf1_95fb_4033_8a3a_bc8d523c80ae.slice/crio-8558a2e04a9f5bd6b8f8706ef7987416e7d3d812f34f70de7eccbd5262ea0ba5 WatchSource:0}: Error finding container 8558a2e04a9f5bd6b8f8706ef7987416e7d3d812f34f70de7eccbd5262ea0ba5: Status 404 returned error can't find the container with id 8558a2e04a9f5bd6b8f8706ef7987416e7d3d812f34f70de7eccbd5262ea0ba5
Jul 13 18:50:12.227400 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:12.227377    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9123bd54_a028_460c_961b_82b16be6168f.slice/crio-7c425ddb8734d26ae0acc2df034c809ed351289d5c7770ad02e580d6f49047e4 WatchSource:0}: Error finding container 7c425ddb8734d26ae0acc2df034c809ed351289d5c7770ad02e580d6f49047e4: Status 404 returned error can't find the container with id 7c425ddb8734d26ae0acc2df034c809ed351289d5c7770ad02e580d6f49047e4
Jul 13 18:50:12.468702 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.468671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:12.468830 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.468700    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:12.468830 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.468723    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:12.468830 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.468738    2569 projected.go:196] Error preparing data for projected volume kube-api-access-dzvfr for pod openshift-network-diagnostics/network-check-target-5kc4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:12.468830 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.468796    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr podName:8d506c0b-8324-498e-962e-6e396a66159e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:13.468778591 +0000 UTC m=+5.574382188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzvfr" (UniqueName: "kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr") pod "network-check-target-5kc4j" (UID: "8d506c0b-8324-498e-962e-6e396a66159e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:12.569842 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.569809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:12.570016 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.569867    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:12.570016 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.569966    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:12.570016 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.569968    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:12.570016 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.570016    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs podName:c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:13.570002509 +0000 UTC m=+5.675606106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs") pod "network-metrics-daemon-pdlbh" (UID: "c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:12.570233 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:12.570034    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret podName:03610f49-0413-408e-9e5c-d1adc6d46204 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:13.570025235 +0000 UTC m=+5.675628832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret") pod "global-pull-secret-syncer-pg7df" (UID: "03610f49-0413-408e-9e5c-d1adc6d46204") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:12.936487 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.936069    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:45:11 +0000 UTC" deadline="2028-01-14 12:05:39.107021816 +0000 UTC"
Jul 13 18:50:12.936487 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.936348    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13193h15m26.170679016s"
Jul 13 18:50:12.990458 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:12.990429    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:13.052239 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.051575    2569 generic.go:356] "Generic (PLEG): container finished" podID="908b719ae1c9201ec6cd9981720d3cfe" containerID="a9e5f44beb286a2dc65e34a504b9e212c0535caee221526caa178eaf63a93a22" exitCode=0
Jul 13 18:50:13.052239 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.051679    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal" event={"ID":"908b719ae1c9201ec6cd9981720d3cfe","Type":"ContainerDied","Data":"a9e5f44beb286a2dc65e34a504b9e212c0535caee221526caa178eaf63a93a22"}
Jul 13 18:50:13.058279 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.058205    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-599db" event={"ID":"103a12f9-a611-452d-ab26-317a59123f8b","Type":"ContainerStarted","Data":"6a501fecf2a607485fc917988e01f1d5370c9d9cb94238f14fcd905ffee7a945"}
Jul 13 18:50:13.059923 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.059841    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z7v8z" event={"ID":"7f27934f-42fa-4166-a6ef-eb8062375c7e","Type":"ContainerStarted","Data":"1001a436612d90ae537bac2ed226257f21aab29f8a9ec73290f471e9dbc566f4"}
Jul 13 18:50:13.062687 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.062611    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8" event={"ID":"9301dc45-e443-459d-a18f-026df166a802","Type":"ContainerStarted","Data":"eb2a37c561092b85ead0513fcbbf1b33f90b4270fb1ef2da6f20c19b8ddaa7d9"}
Jul 13 18:50:13.066341 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.066227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2l425" event={"ID":"7ef7ee3d-6283-41fc-822a-0597c41cdccd","Type":"ContainerStarted","Data":"5f217dbbb9cdea178d18a7a0cb0ff76c7010a3b8636fa7700f680c3b7234130b"}
Jul 13 18:50:13.073479 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.073416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" event={"ID":"9123bd54-a028-460c-961b-82b16be6168f","Type":"ContainerStarted","Data":"7c425ddb8734d26ae0acc2df034c809ed351289d5c7770ad02e580d6f49047e4"}
Jul 13 18:50:13.075118 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.075088    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-864b4" event={"ID":"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae","Type":"ContainerStarted","Data":"8558a2e04a9f5bd6b8f8706ef7987416e7d3d812f34f70de7eccbd5262ea0ba5"}
Jul 13 18:50:13.078283 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.078244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mdpdw" event={"ID":"1927b946-784d-4d94-b311-3eb5824fa9ce","Type":"ContainerStarted","Data":"7ede58d3b2500c4b4b6ab511cfa981722e5e4d196610a21b4757e17c53f6a0a5"}
Jul 13 18:50:13.081453 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.081347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerStarted","Data":"2f497127d5e25de65f0b489543b2c22d00cd1661d25ea36750aeeda3b6a80685"}
Jul 13 18:50:13.083894 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.083833    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bpjbm" event={"ID":"52a0b632-455c-4340-99f2-d4178918e76c","Type":"ContainerStarted","Data":"872883d56085b75af88869cc1e3c8d4f4423e9336566c172ca50b5e3ac809897"}
Jul 13 18:50:13.475715 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.475676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:13.475932 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:13.475879    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:13.475932 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:13.475905    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:13.475932 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:13.475920    2569 projected.go:196] Error preparing data for projected volume kube-api-access-dzvfr for pod openshift-network-diagnostics/network-check-target-5kc4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:13.476092 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:13.475990    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr podName:8d506c0b-8324-498e-962e-6e396a66159e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:15.475969617 +0000 UTC m=+7.581573218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzvfr" (UniqueName: "kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr") pod "network-check-target-5kc4j" (UID: "8d506c0b-8324-498e-962e-6e396a66159e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:13.576777 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.576739    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:13.576956 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.576809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:13.576956 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:13.576942    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:13.577060 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:13.577004    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs podName:c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:15.576986951 +0000 UTC m=+7.682590565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs") pod "network-metrics-daemon-pdlbh" (UID: "c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:13.577427 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:13.577406    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:13.577520 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:13.577466    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret podName:03610f49-0413-408e-9e5c-d1adc6d46204 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:15.57744995 +0000 UTC m=+7.683053561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret") pod "global-pull-secret-syncer-pg7df" (UID: "03610f49-0413-408e-9e5c-d1adc6d46204") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:13.937146 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.937066    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:45:11 +0000 UTC" deadline="2028-04-24 01:04:01.734529964 +0000 UTC"
Jul 13 18:50:13.937146 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:13.937109    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15606h13m47.797425852s"
Jul 13 18:50:14.042943 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:14.042906    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:14.043144 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:14.043043    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:14.043923 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:14.043425    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:14.043923 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:14.043618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:14.043923 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:14.043703    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:14.044232 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:14.043515    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:15.491692 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:15.491653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:15.492137 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:15.491867    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:15.492137 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:15.491888    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:15.492137 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:15.491900    2569 projected.go:196] Error preparing data for projected volume kube-api-access-dzvfr for pod openshift-network-diagnostics/network-check-target-5kc4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:15.492137 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:15.491960    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr podName:8d506c0b-8324-498e-962e-6e396a66159e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:19.491942217 +0000 UTC m=+11.597545828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzvfr" (UniqueName: "kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr") pod "network-check-target-5kc4j" (UID: "8d506c0b-8324-498e-962e-6e396a66159e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:15.593030 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:15.592987    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:15.593187 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:15.593147    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:15.593271 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:15.593231    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:15.593353 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:15.593278    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:15.593353 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:15.593309    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs podName:c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:19.593287616 +0000 UTC m=+11.698891213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs") pod "network-metrics-daemon-pdlbh" (UID: "c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:15.593353 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:15.593332    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret podName:03610f49-0413-408e-9e5c-d1adc6d46204 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:19.593316588 +0000 UTC m=+11.698920189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret") pod "global-pull-secret-syncer-pg7df" (UID: "03610f49-0413-408e-9e5c-d1adc6d46204") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:16.040607 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:16.040572    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:16.040786 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:16.040687    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:16.041050 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:16.040573    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:16.041050 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:16.040997    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:16.041205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:16.041073    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:16.041205 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:16.041165    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:18.041997 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:18.041626    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:18.041997 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:18.041754    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:18.041997 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:18.041875    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:18.041997 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:18.041972    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:18.042653 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:18.042033    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:18.042653 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:18.042137    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:19.528123 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:19.528076    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:19.528635 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:19.528276    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:19.528635 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:19.528303    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:19.528635 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:19.528317    2569 projected.go:196] Error preparing data for projected volume kube-api-access-dzvfr for pod openshift-network-diagnostics/network-check-target-5kc4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:19.528635 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:19.528389    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr podName:8d506c0b-8324-498e-962e-6e396a66159e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:27.528368946 +0000 UTC m=+19.633972559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzvfr" (UniqueName: "kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr") pod "network-check-target-5kc4j" (UID: "8d506c0b-8324-498e-962e-6e396a66159e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:19.628501 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:19.628447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:19.628694 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:19.628552    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:19.628694 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:19.628689    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:19.628829 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:19.628715    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:19.628829 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:19.628749    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret podName:03610f49-0413-408e-9e5c-d1adc6d46204 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:27.628731508 +0000 UTC m=+19.734335111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret") pod "global-pull-secret-syncer-pg7df" (UID: "03610f49-0413-408e-9e5c-d1adc6d46204") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:19.628829 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:19.628780    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs podName:c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:27.62876195 +0000 UTC m=+19.734365550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs") pod "network-metrics-daemon-pdlbh" (UID: "c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:20.042065 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:20.041627    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:20.042065 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:20.041753    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:20.042065 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:20.042032    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:20.042377 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:20.042147    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:20.042377 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:20.042217    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:20.042377 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:20.042312    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:22.041293 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:22.041075    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:22.041293 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:22.041108    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:22.041293 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:22.041075    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:22.041293 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:22.041205    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:22.041820 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:22.041342    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:22.041820 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:22.041431    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:24.040728 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:24.040692    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:24.041157 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:24.040692    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:24.041157 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:24.040835    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:24.041157 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:24.040692    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:24.041157 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:24.040932    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:24.041157 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:24.041053    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:26.041144 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:26.041110    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:26.041144 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:26.041149    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:26.041662 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:26.041121    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:26.041662 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:26.041237    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:26.041662 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:26.041333    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:26.041662 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:26.041424    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:27.589581 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:27.589542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:27.590078 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:27.589709    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:27.590078 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:27.589731    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:27.590078 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:27.589745    2569 projected.go:196] Error preparing data for projected volume kube-api-access-dzvfr for pod openshift-network-diagnostics/network-check-target-5kc4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:27.590078 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:27.589836    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr podName:8d506c0b-8324-498e-962e-6e396a66159e nodeName:}" failed. No retries permitted until 2026-07-13 18:50:43.589804739 +0000 UTC m=+35.695408354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzvfr" (UniqueName: "kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr") pod "network-check-target-5kc4j" (UID: "8d506c0b-8324-498e-962e-6e396a66159e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:27.690261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:27.690218    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:27.690451 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:27.690274    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:27.690451 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:27.690377    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:27.690451 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:27.690378    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:27.690451 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:27.690443    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs podName:c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:43.690425228 +0000 UTC m=+35.796028825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs") pod "network-metrics-daemon-pdlbh" (UID: "c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:27.690694 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:27.690458    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret podName:03610f49-0413-408e-9e5c-d1adc6d46204 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:43.690451815 +0000 UTC m=+35.796055412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret") pod "global-pull-secret-syncer-pg7df" (UID: "03610f49-0413-408e-9e5c-d1adc6d46204") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:28.040488 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:28.040419    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:28.040488 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:28.040434    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:28.040749 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:28.040568    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:28.040749 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:28.040652    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:28.040749 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:28.040682    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:28.040888 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:28.040797    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:30.041913 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:30.041875    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:30.042321 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:30.041970    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:30.042321 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:30.042000    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:30.042321 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:30.042053    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:30.042321 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:30.042071    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:30.042321 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:30.042170    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:32.040704 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.040671    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:32.041223 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.040711    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:32.041223 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.040672    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:32.041223 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:32.040808    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:32.041223 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:32.040885    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:32.041223 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:32.040940    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:32.124889 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.124868    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 18:50:32.125266 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.125242    2569 generic.go:356] "Generic (PLEG): container finished" podID="7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7" containerID="0a69ff083435688820a2948cfca87b2cbc7913907b188d9d29ba7ad34371b6b5" exitCode=1
Jul 13 18:50:32.125359 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.125302    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerDied","Data":"0a69ff083435688820a2948cfca87b2cbc7913907b188d9d29ba7ad34371b6b5"}
Jul 13 18:50:32.125423 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.125378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerStarted","Data":"5b5bf694db021683e150d54a17ab6acc2e8539097a0537f8b8248a81fb1d901c"}
Jul 13 18:50:32.127105 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.127066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bpjbm" event={"ID":"52a0b632-455c-4340-99f2-d4178918e76c","Type":"ContainerStarted","Data":"599c8d56ada1442415eb783c7f849697addc4f42c2bd5c6eea628c91fbc8df80"}
Jul 13 18:50:32.129125 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.129096    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal" event={"ID":"908b719ae1c9201ec6cd9981720d3cfe","Type":"ContainerStarted","Data":"3c8c03af18040b24c79bcc8adeb040972963ca77090a39d57393ebf3b130ec35"}
Jul 13 18:50:32.130722 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.130695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal" event={"ID":"cfc6c26c84bf408e478864f003148e2a","Type":"ContainerStarted","Data":"61905bcfc3fb475038830db955c83c8c0a2a088f513c3a2ccf6549797865b00e"}
Jul 13 18:50:32.132808 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.132780    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-599db" event={"ID":"103a12f9-a611-452d-ab26-317a59123f8b","Type":"ContainerStarted","Data":"98af43a93d6786b66d693fe4fd861cca8dead3447654c32abcb7e96732bf8bc3"}
Jul 13 18:50:32.134499 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.134471    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z7v8z" event={"ID":"7f27934f-42fa-4166-a6ef-eb8062375c7e","Type":"ContainerStarted","Data":"8896b0f6e180947167b210b330dda2a8d6341fc20b828e3f5b44077cfb7bf7ff"}
Jul 13 18:50:32.136170 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.136147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8" event={"ID":"9301dc45-e443-459d-a18f-026df166a802","Type":"ContainerStarted","Data":"c48faf88ffc8d4551fed2bc2c814391ee2e558974af8cd3f058cb697431b0ae0"}
Jul 13 18:50:32.137980 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.137951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2l425" event={"ID":"7ef7ee3d-6283-41fc-822a-0597c41cdccd","Type":"ContainerStarted","Data":"ef12abc81050a7a7c97651c01d26b0ed3425b74434e31000b94043bb82ed40a3"}
Jul 13 18:50:32.139632 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.139605    2569 generic.go:356] "Generic (PLEG): container finished" podID="9123bd54-a028-460c-961b-82b16be6168f" containerID="504cd2dd9981289c4ca92c973124e22d900994d33dcceabb911373622261ced5" exitCode=0
Jul 13 18:50:32.139724 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.139687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" event={"ID":"9123bd54-a028-460c-961b-82b16be6168f","Type":"ContainerDied","Data":"504cd2dd9981289c4ca92c973124e22d900994d33dcceabb911373622261ced5"}
Jul 13 18:50:32.145010 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.144983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-864b4" event={"ID":"bd43cbf1-95fb-4033-8a3a-bc8d523c80ae","Type":"ContainerStarted","Data":"3bfa9cd4c8bf5dfd5f74595915fbb543d0e03fdb399f6f05e2f866a0ac2cab84"}
Jul 13 18:50:32.152776 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.152700    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bpjbm" podStartSLOduration=2.981416141 podStartE2EDuration="22.152648511s" totalImagesPullingTime="19.17123237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:32.151266179 +0000 UTC m=+24.256869799" watchObservedRunningTime="2026-07-13 18:50:32.152648511 +0000 UTC m=+24.258252134"
Jul 13 18:50:32.175425 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.175370    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-864b4" podStartSLOduration=2.419303634 podStartE2EDuration="22.175346964s" totalImagesPullingTime="19.75604333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:32.174128826 +0000 UTC m=+24.279732446" watchObservedRunningTime="2026-07-13 18:50:32.175346964 +0000 UTC m=+24.280950586"
Jul 13 18:50:32.214676 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.214622    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-96.ec2.internal" podStartSLOduration=21.214598906 podStartE2EDuration="21.214598906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:32.191427292 +0000 UTC m=+24.297030912" watchObservedRunningTime="2026-07-13 18:50:32.214598906 +0000 UTC m=+24.320202528"
Jul 13 18:50:32.214763 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.214729    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-599db" podStartSLOduration=3.052168981 podStartE2EDuration="22.214720073s" totalImagesPullingTime="19.162551092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:32.214352918 +0000 UTC m=+24.319956537" watchObservedRunningTime="2026-07-13 18:50:32.214720073 +0000 UTC m=+24.320323693"
Jul 13 18:50:32.265303 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.265258    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z7v8z" podStartSLOduration=2.786973451 podStartE2EDuration="22.26524051s" totalImagesPullingTime="19.478267059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:32.23822539 +0000 UTC m=+24.343829011" watchObservedRunningTime="2026-07-13 18:50:32.26524051 +0000 UTC m=+24.370844129"
Jul 13 18:50:32.288389 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:32.288341    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2l425" podStartSLOduration=2.819350976 podStartE2EDuration="22.288325001s" totalImagesPullingTime="19.468974025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:32.287613682 +0000 UTC m=+24.393217302" watchObservedRunningTime="2026-07-13 18:50:32.288325001 +0000 UTC m=+24.393928619"
Jul 13 18:50:33.064151 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.064112    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:33.148706 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.148664    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mdpdw" event={"ID":"1927b946-784d-4d94-b311-3eb5824fa9ce","Type":"ContainerStarted","Data":"9a49d48c29f572d18b96b835482989a59c8d7a97ce097293128b630fd6fc47ec"}
Jul 13 18:50:33.152932 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.152909    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 18:50:33.153461 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.153431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerStarted","Data":"78d467ba9c0547f86deb2d7de312477fbaf63b72bb6b9b302bc79ae6e6d71fb8"}
Jul 13 18:50:33.153592 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.153510    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerStarted","Data":"4f1722c3989d9b4a86b6eaa65a82562c4eb3572a95bd4b80f61a0a750d9bb909"}
Jul 13 18:50:33.153592 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.153579    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerStarted","Data":"e7bc8f1c48c0070553515ee6000beeedb8039a544c44e3003c9cf507e89bfca4"}
Jul 13 18:50:33.153706 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.153628    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerStarted","Data":"bc9ea75cae29ee27651d19301f74a8bdc0510934747bff3f2b7a32d036791703"}
Jul 13 18:50:33.168459 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.168407    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mdpdw" podStartSLOduration=4.00371008 podStartE2EDuration="23.168389994s" totalImagesPullingTime="19.164679914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:33.168180591 +0000 UTC m=+25.273784210" watchObservedRunningTime="2026-07-13 18:50:33.168389994 +0000 UTC m=+25.273993615"
Jul 13 18:50:33.168900 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.168856    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-96.ec2.internal" podStartSLOduration=22.168843251 podStartE2EDuration="22.168843251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:32.311710888 +0000 UTC m=+24.417314508" watchObservedRunningTime="2026-07-13 18:50:33.168843251 +0000 UTC m=+25.274446883"
Jul 13 18:50:33.201041 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.201016    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 18:50:33.243594 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.243562    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:33.347240 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.347159    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:33.447957 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.447909    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:33.575351 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.575315    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:33.914691 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.914570    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T18:50:33.201035357Z","UUID":"da9ac68b-69d3-493b-8243-31919c982f55","Handler":null,"Name":"","Endpoint":""}
Jul 13 18:50:33.916150 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.916122    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 18:50:33.916274 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.916169    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 18:50:33.967847 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:33.967813    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:34.040554 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:34.040500    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:34.040736 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:34.040657    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:34.040736 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:34.040512    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:34.040736 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:34.040500    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:34.040908 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:34.040755    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:34.040908 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:34.040825    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:34.158177 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:34.158139    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8" event={"ID":"9301dc45-e443-459d-a18f-026df166a802","Type":"ContainerStarted","Data":"23a0c3be2c59fef153354a821e630506c261dd5e7915915744c07dc4981ee4e8"}
Jul 13 18:50:34.356903 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:34.356873    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:35.163513 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:35.163290    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8" event={"ID":"9301dc45-e443-459d-a18f-026df166a802","Type":"ContainerStarted","Data":"16f2529ce79eaa315462fcfc1e77ad129db066b894457e409f2c2e4d42eaab5c"}
Jul 13 18:50:35.168845 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:35.168788    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:35.169136 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:35.169120    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 18:50:35.169643 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:35.169615    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerStarted","Data":"5cd243c22e272bbbf373f16061c8c4dd6832bf8175629272208fe3df6bdd3f46"}
Jul 13 18:50:35.184572 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:35.184500    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs9b8" podStartSLOduration=3.58235344 podStartE2EDuration="25.184487004s" totalImagesPullingTime="21.602133564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:35.184222582 +0000 UTC m=+27.289826200" watchObservedRunningTime="2026-07-13 18:50:35.184487004 +0000 UTC m=+27.290090623"
Jul 13 18:50:35.510332 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:35.510248    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:35.612878 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:35.612844    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:35.766368 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:35.766276    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:35.949716 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:35.949683    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:36.041277 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:36.041201    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:36.041277 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:36.041214    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:36.041277 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:36.041237    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:36.041564 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:36.041362    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:36.041564 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:36.041445    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:36.041564 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:36.041511    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:36.151890 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:36.151858    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:36.752378 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:36.752348    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:36.953628 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:36.953362    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:36.954415 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:36.954398    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:36.994687 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:36.994660    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:37.177717 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.177693    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 18:50:37.178126 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.178101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerStarted","Data":"55ccfae61400418d2390fd3d719e23f61fbdb4a6fb3d1ec78291cdc5b718ccce"}
Jul 13 18:50:37.178374 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.178346    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:37.178522 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.178431    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:37.178599 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.178546    2569 scope.go:117] "RemoveContainer" containerID="0a69ff083435688820a2948cfca87b2cbc7913907b188d9d29ba7ad34371b6b5"
Jul 13 18:50:37.180119 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.180094    2569 generic.go:356] "Generic (PLEG): container finished" podID="9123bd54-a028-460c-961b-82b16be6168f" containerID="1f98617389e903761bef973b29cb05e9dc05d7b80747ee4242325c1cc8622dbf" exitCode=0
Jul 13 18:50:37.180214 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.180172    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" event={"ID":"9123bd54-a028-460c-961b-82b16be6168f","Type":"ContainerDied","Data":"1f98617389e903761bef973b29cb05e9dc05d7b80747ee4242325c1cc8622dbf"}
Jul 13 18:50:37.180438 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.180422    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:37.181068 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.180998    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2l425"
Jul 13 18:50:37.195543 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.195511    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:37.342318 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.342283    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:37.571915 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.571835    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:37.750541 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.750488    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:37.943278 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:37.943250    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:38.040260 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:38.040223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:38.040260 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:38.040244    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:38.040560 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:38.040223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:38.040560 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:38.040347    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:38.040560 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:38.040406    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:38.040560 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:38.040478    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:38.158136 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:38.158099    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:38.185655 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:38.185635    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 18:50:38.185966 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:38.185944    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" event={"ID":"7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7","Type":"ContainerStarted","Data":"e015bc3bf2c244aba82df194815884004526bd6aaf857e7d0e74403e8cad4c98"}
Jul 13 18:50:38.188999 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:38.188947    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:38.206418 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:38.206378    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:50:38.231600 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:38.231558    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq" podStartSLOduration=8.704937138 podStartE2EDuration="28.231546452s" totalImagesPullingTime="19.526609314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:38.229972701 +0000 UTC m=+30.335576354" watchObservedRunningTime="2026-07-13 18:50:38.231546452 +0000 UTC m=+30.337150062"
Jul 13 18:50:39.150205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:39.150176    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:39.190342 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:39.190302    2569 generic.go:356] "Generic (PLEG): container finished" podID="9123bd54-a028-460c-961b-82b16be6168f" containerID="6c6e537d6fac02466756da3841185988fa1da8c23ad2b9d0cb7118ff5fa251c1" exitCode=0
Jul 13 18:50:39.190518 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:39.190349    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" event={"ID":"9123bd54-a028-460c-961b-82b16be6168f","Type":"ContainerDied","Data":"6c6e537d6fac02466756da3841185988fa1da8c23ad2b9d0cb7118ff5fa251c1"}
Jul 13 18:50:39.366578 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:39.366538    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:39.549183 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:39.549101    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:39.746917 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:39.746887    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:39.987777 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:39.987748    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:40.041487 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:40.041463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:40.041626 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:40.041562    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:40.041679 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:40.041640    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:40.041748 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:40.041724    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:40.041794 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:40.041755    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:40.041849 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:40.041833    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:40.145790 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:40.145762    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:40.356310 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:40.356288    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:40.763587 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:40.763555    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:41.196986 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:41.196953    2569 generic.go:356] "Generic (PLEG): container finished" podID="9123bd54-a028-460c-961b-82b16be6168f" containerID="7d346012258441f92ac113921657ffb1fa405e81e9436f9457792d78e7c2ca88" exitCode=0
Jul 13 18:50:41.197143 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:41.197014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" event={"ID":"9123bd54-a028-460c-961b-82b16be6168f","Type":"ContainerDied","Data":"7d346012258441f92ac113921657ffb1fa405e81e9436f9457792d78e7c2ca88"}
Jul 13 18:50:41.339400 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:41.339364    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:41.463658 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:41.463579    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:41.565502 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:41.565470    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:41.632915 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:41.632883    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:41.742582 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:41.742483    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:41.945825 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:41.945792    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:42.040541 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:42.040441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:42.040713 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:42.040448    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:42.040713 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:42.040596    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:42.040713 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:42.040675    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:42.040713 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:42.040449    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:42.040913 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:42.040775    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:42.161946 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:42.161909    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:43.061413 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:43.061377    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:43.144255 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:43.144222    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:43.343907 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:43.343826    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:43.553699 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:43.553662    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:43.601650 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:43.601581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:43.601803 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:43.601772    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:43.601803 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:43.601794    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:43.601908 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:43.601807    2569 projected.go:196] Error preparing data for projected volume kube-api-access-dzvfr for pod openshift-network-diagnostics/network-check-target-5kc4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:43.601908 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:43.601876    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr podName:8d506c0b-8324-498e-962e-6e396a66159e nodeName:}" failed. No retries permitted until 2026-07-13 18:51:15.601856554 +0000 UTC m=+67.707460174 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzvfr" (UniqueName: "kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr") pod "network-check-target-5kc4j" (UID: "8d506c0b-8324-498e-962e-6e396a66159e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:43.702230 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:43.702196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:43.702413 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:43.702250    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:43.702413 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:43.702365    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:43.702508 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:43.702423    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs podName:c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:15.702407634 +0000 UTC m=+67.808011230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs") pod "network-metrics-daemon-pdlbh" (UID: "c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:43.702508 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:43.702365    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:43.702508 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:43.702489    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret podName:03610f49-0413-408e-9e5c-d1adc6d46204 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:15.702478345 +0000 UTC m=+67.808081960 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret") pod "global-pull-secret-syncer-pg7df" (UID: "03610f49-0413-408e-9e5c-d1adc6d46204") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:43.746437 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:43.746406    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:44.040377 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:44.040347    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:44.040576 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:44.040347    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:44.040576 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:44.040475    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:44.040576 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:44.040565    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:44.040749 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:44.040346    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:44.040749 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:44.040657    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:45.208938 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:45.208905    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" event={"ID":"9123bd54-a028-460c-961b-82b16be6168f","Type":"ContainerStarted","Data":"400d6b2eac4079d105cf5475ea3793920e04af461bc90d74498e6dad3f53d5a3"}
Jul 13 18:50:46.040210 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:46.040178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:46.040377 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:46.040186    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:46.040377 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:46.040284    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:46.040377 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:46.040189    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:46.040474 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:46.040350    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:46.040474 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:46.040437    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:46.213709 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:46.213679    2569 generic.go:356] "Generic (PLEG): container finished" podID="9123bd54-a028-460c-961b-82b16be6168f" containerID="400d6b2eac4079d105cf5475ea3793920e04af461bc90d74498e6dad3f53d5a3" exitCode=0
Jul 13 18:50:46.214054 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:46.213722    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" event={"ID":"9123bd54-a028-460c-961b-82b16be6168f","Type":"ContainerDied","Data":"400d6b2eac4079d105cf5475ea3793920e04af461bc90d74498e6dad3f53d5a3"}
Jul 13 18:50:47.219049 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:47.218860    2569 generic.go:356] "Generic (PLEG): container finished" podID="9123bd54-a028-460c-961b-82b16be6168f" containerID="3646470426d1e4a52139015f5448dcbceeee0dfff0326934758a0b2af626fe3e" exitCode=0
Jul 13 18:50:47.219407 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:47.218906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" event={"ID":"9123bd54-a028-460c-961b-82b16be6168f","Type":"ContainerDied","Data":"3646470426d1e4a52139015f5448dcbceeee0dfff0326934758a0b2af626fe3e"}
Jul 13 18:50:48.041092 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:48.041062    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:48.041290 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:48.041062    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:48.041290 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:48.041176    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:48.041290 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:48.041191    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:48.041290 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:48.041267    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:48.041506 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:48.041354    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:48.224751 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:48.224722    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" event={"ID":"9123bd54-a028-460c-961b-82b16be6168f","Type":"ContainerStarted","Data":"911fd77b5e5604a311ffe801d20fe21e9e35573996a2f981cddff09f05bba5b7"}
Jul 13 18:50:48.250415 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:48.250368    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nkdxv" podStartSLOduration=8.540451225 podStartE2EDuration="38.250350467s" totalImagesPullingTime="29.709899242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:48.249753503 +0000 UTC m=+40.355357121" watchObservedRunningTime="2026-07-13 18:50:48.250350467 +0000 UTC m=+40.355954089"
Jul 13 18:50:50.041567 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:50.041517    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:50.041952 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:50.041574    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:50.041952 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:50.041636    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:50.041952 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:50.041682    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:50.041952 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:50.041717    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:50.041952 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:50.041784    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:52.040438 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:52.040403    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:52.041053 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:52.040405    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:52.041053 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:52.040519    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:52.041053 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:52.040598    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:52.041053 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:52.040404    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:52.041053 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:52.040667    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:53.241350 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:53.241313    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:53.419573 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:53.419544    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:53.507223 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:53.507149    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:53.605778 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:53.605742    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:53.723694 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:53.723659    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:53.832414 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:53.832350    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:54.041180 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.041152    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:54.041387 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.041154    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:54.041387 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:54.041303    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:54.041387 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.041353    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:54.041614 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:54.041582    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:54.041689 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:54.041628    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:54.144010 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.143979    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:54.225266 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.225234    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:54.341593 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.341558    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:54.484474 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.484397    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:54.665504 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.665470    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:54.779174 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.779149    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:54.997036 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.996996    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdlbh"]
Jul 13 18:50:54.997197 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.997137    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:54.997315 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:54.997290    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:54.997869 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.997847    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5kc4j"]
Jul 13 18:50:54.998026 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:54.998007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:54.998130 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:54.998110    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:55.002950 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:55.002925    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pg7df"]
Jul 13 18:50:55.003060 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:55.003006    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:55.003128 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:55.003099    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:55.152001 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:55.151971    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:55.273823 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:55.273763    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:55.417162 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:55.417135    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:55.517594 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:55.517554    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:55.700642 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:55.700549    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:55.844243 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:55.844197    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:55.982737 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:55.982646    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:56.040939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:56.040911    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:56.041096 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:56.041040    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:56.509715 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:56.509674    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:56.906197 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:56.906166    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:56.976393 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:56.976355    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:57.040583 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:57.040553    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:57.040747 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:57.040553    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:57.040747 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:57.040658    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5kc4j" podUID="8d506c0b-8324-498e-962e-6e396a66159e"
Jul 13 18:50:57.040747 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:57.040728    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg7df" podUID="03610f49-0413-408e-9e5c-d1adc6d46204"
Jul 13 18:50:58.040943 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.040909    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:50:58.041314 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:50:58.041049    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlbh" podUID="c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84"
Jul 13 18:50:58.691238 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.691170    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-96.ec2.internal" event="NodeReady"
Jul 13 18:50:58.691376 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.691283    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 18:50:58.747476 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.747449    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tz6g6"]
Jul 13 18:50:58.758339 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.758318    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ngb8f"]
Jul 13 18:50:58.758457 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.758447    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.761480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.761441    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 18:50:58.761480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.761458    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zz2bf\""
Jul 13 18:50:58.761480 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.761440    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 18:50:58.770411 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.770391    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tz6g6"]
Jul 13 18:50:58.770508 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.770431    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngb8f"]
Jul 13 18:50:58.770508 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.770494    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.773920 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.773902    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 18:50:58.774031 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.773939    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 18:50:58.774031 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.773904    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 18:50:58.774031 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.773959    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-22w5p\""
Jul 13 18:50:58.774291 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.774274    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 18:50:58.778460 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.778441    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pqhbw"]
Jul 13 18:50:58.793446 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.793428    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqhbw"
Jul 13 18:50:58.796351 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.796329    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 18:50:58.796453 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.796389    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 18:50:58.796723 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.796703    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 18:50:58.796829 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.796740    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqhbw"]
Jul 13 18:50:58.799323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.799304    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-96xp4\""
Jul 13 18:50:58.814889 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.814865    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7ft\" (UniqueName: \"kubernetes.io/projected/ec484b05-f937-4909-a863-02e7faa131f9-kube-api-access-qm7ft\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.815090 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.814922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/185e1d28-22e4-4b9f-bf9a-c49a8169e375-crio-socket\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.815090 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.814948    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/185e1d28-22e4-4b9f-bf9a-c49a8169e375-data-volume\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.815090 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.814975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/185e1d28-22e4-4b9f-bf9a-c49a8169e375-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.815090 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.815023    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rdl2\" (UniqueName: \"kubernetes.io/projected/185e1d28-22e4-4b9f-bf9a-c49a8169e375-kube-api-access-4rdl2\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.815090 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.815072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec484b05-f937-4909-a863-02e7faa131f9-tmp-dir\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.815318 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.815124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/185e1d28-22e4-4b9f-bf9a-c49a8169e375-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.815318 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.815147    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4m4\" (UniqueName: \"kubernetes.io/projected/54ed57eb-2265-4c86-9ca9-fbbaa753706f-kube-api-access-lh4m4\") pod \"ingress-canary-pqhbw\" (UID: \"54ed57eb-2265-4c86-9ca9-fbbaa753706f\") " pod="openshift-ingress-canary/ingress-canary-pqhbw"
Jul 13 18:50:58.815318 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.815172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec484b05-f937-4909-a863-02e7faa131f9-metrics-tls\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.815318 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.815192    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ed57eb-2265-4c86-9ca9-fbbaa753706f-cert\") pod \"ingress-canary-pqhbw\" (UID: \"54ed57eb-2265-4c86-9ca9-fbbaa753706f\") " pod="openshift-ingress-canary/ingress-canary-pqhbw"
Jul 13 18:50:58.815318 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.815217    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec484b05-f937-4909-a863-02e7faa131f9-config-volume\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.915521 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.915484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rdl2\" (UniqueName: \"kubernetes.io/projected/185e1d28-22e4-4b9f-bf9a-c49a8169e375-kube-api-access-4rdl2\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.915521 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.915541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec484b05-f937-4909-a863-02e7faa131f9-tmp-dir\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.915738 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.915579    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/185e1d28-22e4-4b9f-bf9a-c49a8169e375-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.915738 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.915596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4m4\" (UniqueName: \"kubernetes.io/projected/54ed57eb-2265-4c86-9ca9-fbbaa753706f-kube-api-access-lh4m4\") pod \"ingress-canary-pqhbw\" (UID: \"54ed57eb-2265-4c86-9ca9-fbbaa753706f\") " pod="openshift-ingress-canary/ingress-canary-pqhbw"
Jul 13 18:50:58.915738 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.915615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec484b05-f937-4909-a863-02e7faa131f9-metrics-tls\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.915738 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.915641    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ed57eb-2265-4c86-9ca9-fbbaa753706f-cert\") pod \"ingress-canary-pqhbw\" (UID: \"54ed57eb-2265-4c86-9ca9-fbbaa753706f\") " pod="openshift-ingress-canary/ingress-canary-pqhbw"
Jul 13 18:50:58.915738 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.915669    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec484b05-f937-4909-a863-02e7faa131f9-config-volume\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.916000 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.915975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7ft\" (UniqueName: \"kubernetes.io/projected/ec484b05-f937-4909-a863-02e7faa131f9-kube-api-access-qm7ft\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.916059 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.915978    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec484b05-f937-4909-a863-02e7faa131f9-tmp-dir\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.916372 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.916035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/185e1d28-22e4-4b9f-bf9a-c49a8169e375-crio-socket\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.916372 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.916153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/185e1d28-22e4-4b9f-bf9a-c49a8169e375-data-volume\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.916372 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.916185    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/185e1d28-22e4-4b9f-bf9a-c49a8169e375-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.916372 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.916187    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/185e1d28-22e4-4b9f-bf9a-c49a8169e375-crio-socket\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.916372 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.916317    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/185e1d28-22e4-4b9f-bf9a-c49a8169e375-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.916616 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.916390    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec484b05-f937-4909-a863-02e7faa131f9-config-volume\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.916616 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.916399    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/185e1d28-22e4-4b9f-bf9a-c49a8169e375-data-volume\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.919745 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.919717    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/185e1d28-22e4-4b9f-bf9a-c49a8169e375-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.919745 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.919729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec484b05-f937-4909-a863-02e7faa131f9-metrics-tls\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.919882 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.919807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54ed57eb-2265-4c86-9ca9-fbbaa753706f-cert\") pod \"ingress-canary-pqhbw\" (UID: \"54ed57eb-2265-4c86-9ca9-fbbaa753706f\") " pod="openshift-ingress-canary/ingress-canary-pqhbw"
Jul 13 18:50:58.924870 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.924849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rdl2\" (UniqueName: \"kubernetes.io/projected/185e1d28-22e4-4b9f-bf9a-c49a8169e375-kube-api-access-4rdl2\") pod \"insights-runtime-extractor-ngb8f\" (UID: \"185e1d28-22e4-4b9f-bf9a-c49a8169e375\") " pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:58.925101 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.925082    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7ft\" (UniqueName: \"kubernetes.io/projected/ec484b05-f937-4909-a863-02e7faa131f9-kube-api-access-qm7ft\") pod \"dns-default-tz6g6\" (UID: \"ec484b05-f937-4909-a863-02e7faa131f9\") " pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:58.925171 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.925149    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4m4\" (UniqueName: \"kubernetes.io/projected/54ed57eb-2265-4c86-9ca9-fbbaa753706f-kube-api-access-lh4m4\") pod \"ingress-canary-pqhbw\" (UID: \"54ed57eb-2265-4c86-9ca9-fbbaa753706f\") " pod="openshift-ingress-canary/ingress-canary-pqhbw"
Jul 13 18:50:58.942312 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:58.942260    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:59.040794 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.040763    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:50:59.040918 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.040800    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:50:59.043513 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.043484    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 18:50:59.043861 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.043545    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 18:50:59.043861 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.043554    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nk8z7\""
Jul 13 18:50:59.043861 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.043822    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:50:59.050696 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.050675    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:50:59.068546 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.068513    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:50:59.078698 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.078681    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngb8f"
Jul 13 18:50:59.102202 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.102180    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqhbw"
Jul 13 18:50:59.272752 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.272717    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqhbw"]
Jul 13 18:50:59.273677 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.273650    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tz6g6"]
Jul 13 18:50:59.284488 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:50:59.284463    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngb8f"]
Jul 13 18:50:59.288503 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:50:59.288481    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185e1d28_22e4_4b9f_bf9a_c49a8169e375.slice/crio-b110f4e2cb732c675e48f9011510ef6ec46f6c13981cbec0281bc2a95a1dcb3a WatchSource:0}: Error finding container b110f4e2cb732c675e48f9011510ef6ec46f6c13981cbec0281bc2a95a1dcb3a: Status 404 returned error can't find the container with id b110f4e2cb732c675e48f9011510ef6ec46f6c13981cbec0281bc2a95a1dcb3a
Jul 13 18:51:00.044927 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.044891    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:51:00.048816 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.048620    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfqjl\""
Jul 13 18:51:00.049669 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.049648    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:51:00.215683 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.215610    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:00.255434 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.255395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngb8f" event={"ID":"185e1d28-22e4-4b9f-bf9a-c49a8169e375","Type":"ContainerStarted","Data":"b110f4e2cb732c675e48f9011510ef6ec46f6c13981cbec0281bc2a95a1dcb3a"}
Jul 13 18:51:00.257180 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.257151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tz6g6" event={"ID":"ec484b05-f937-4909-a863-02e7faa131f9","Type":"ContainerStarted","Data":"c0f6957a5b1d4170b29883decaa99380f97850984932670bbad9f927222d5d0f"}
Jul 13 18:51:00.259152 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.259123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqhbw" event={"ID":"54ed57eb-2265-4c86-9ca9-fbbaa753706f","Type":"ContainerStarted","Data":"535cec787f2943902ec5d08b948ff50cde69ebfbbeeda47481aa2f914b0cf803"}
Jul 13 18:51:00.336495 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.336362    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:00.493605 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.493570    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:00.582587 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:00.582549    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:01.166362 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.166312    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:01.237294 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.237257    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:01.517592 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.517473    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:01.571271 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.571234    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6vlsp"]
Jul 13 18:51:01.583949 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.583920    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.588308 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.588284    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 18:51:01.588571 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.588552    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 18:51:01.588697 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.588606    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 13 18:51:01.588697 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.588625    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lf76f\""
Jul 13 18:51:01.588808 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.588552    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 13 18:51:01.590519 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.590496    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 18:51:01.597583 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.597559    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6vlsp"]
Jul 13 18:51:01.621696 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.621669    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:01.637059 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.637029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/83ae5ff5-1970-4f25-99f4-c53c7f19d710-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.637059 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.637064    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83ae5ff5-1970-4f25-99f4-c53c7f19d710-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.637275 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.637082    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkd5\" (UniqueName: \"kubernetes.io/projected/83ae5ff5-1970-4f25-99f4-c53c7f19d710-kube-api-access-ntkd5\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.637275 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.637201    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83ae5ff5-1970-4f25-99f4-c53c7f19d710-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.738472 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.738432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83ae5ff5-1970-4f25-99f4-c53c7f19d710-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.738677 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.738505    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/83ae5ff5-1970-4f25-99f4-c53c7f19d710-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.738677 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.738552    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83ae5ff5-1970-4f25-99f4-c53c7f19d710-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.738677 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.738580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkd5\" (UniqueName: \"kubernetes.io/projected/83ae5ff5-1970-4f25-99f4-c53c7f19d710-kube-api-access-ntkd5\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.738677 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:51:01.738656    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 13 18:51:01.738885 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:51:01.738726    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83ae5ff5-1970-4f25-99f4-c53c7f19d710-prometheus-operator-tls podName:83ae5ff5-1970-4f25-99f4-c53c7f19d710 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:02.238706185 +0000 UTC m=+54.344309786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/83ae5ff5-1970-4f25-99f4-c53c7f19d710-prometheus-operator-tls") pod "prometheus-operator-64785b6675-6vlsp" (UID: "83ae5ff5-1970-4f25-99f4-c53c7f19d710") : secret "prometheus-operator-tls" not found
Jul 13 18:51:01.739544 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.739500    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83ae5ff5-1970-4f25-99f4-c53c7f19d710-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.742436 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.742230    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83ae5ff5-1970-4f25-99f4-c53c7f19d710-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.748092 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.748067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkd5\" (UniqueName: \"kubernetes.io/projected/83ae5ff5-1970-4f25-99f4-c53c7f19d710-kube-api-access-ntkd5\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:01.830834 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.830721    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:01.947690 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:01.947654    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.061155 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:02.061124    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.148009 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:02.147978    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.243761 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:02.243728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/83ae5ff5-1970-4f25-99f4-c53c7f19d710-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:02.246464 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:02.246442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/83ae5ff5-1970-4f25-99f4-c53c7f19d710-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6vlsp\" (UID: \"83ae5ff5-1970-4f25-99f4-c53c7f19d710\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:02.366124 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:02.366091    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.450747 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:02.450661    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.494576 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:02.494541    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp"
Jul 13 18:51:02.810094 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:02.810017    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:02.916955 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:02.916924    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:03.569738 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:03.569708    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:03.672947 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:03.672914    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:03.986302 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:03.986263    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:04.085523 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.085491    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:04.349216 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.349179    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bbdfd99cd-r9prl"]
Jul 13 18:51:04.361633 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.361605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.365766 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.365740    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 13 18:51:04.365898 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.365788    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 13 18:51:04.365898 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.365815    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 13 18:51:04.366414 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.366388    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 13 18:51:04.366550 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.366440    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 13 18:51:04.366550 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.366511    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 13 18:51:04.366677 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.366560    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 13 18:51:04.366677 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.366589    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mzsrw\""
Jul 13 18:51:04.370653 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.370272    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbdfd99cd-r9prl"]
Jul 13 18:51:04.461862 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.461826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-oauth-config\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.462029 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.461872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-oauth-serving-cert\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.462029 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.461981    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhp2\" (UniqueName: \"kubernetes.io/projected/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-kube-api-access-rvhp2\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.462029 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.462017    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-config\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.462183 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.462127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-service-ca\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.462183 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.462160    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-serving-cert\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.563113 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.563036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-service-ca\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.563113 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.563086    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-serving-cert\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.563737 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.563128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-oauth-config\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.563737 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.563156    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-oauth-serving-cert\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.563737 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.563199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhp2\" (UniqueName: \"kubernetes.io/projected/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-kube-api-access-rvhp2\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.563737 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.563225    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-config\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.563978 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.563840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-service-ca\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.563978 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.563909    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-config\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.564083 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.563994    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-oauth-serving-cert\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.566144 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.566103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-oauth-config\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.566582 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.566550    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-serving-cert\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.577225 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.577169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhp2\" (UniqueName: \"kubernetes.io/projected/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-kube-api-access-rvhp2\") pod \"console-bbdfd99cd-r9prl\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") " pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.674043 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.674000    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:04.778393 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.778355    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:04.856760 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:04.856687    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:05.195251 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:05.195221    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:05.315848 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:05.315569    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:05.329457 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:05.329385    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6vlsp"]
Jul 13 18:51:05.336512 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:05.336445    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbdfd99cd-r9prl"]
Jul 13 18:51:05.337376 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:05.337344    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ae5ff5_1970_4f25_99f4_c53c7f19d710.slice/crio-008eaec48c6be2df5076145f9b110c20db798e1cf64d18aabc701d826a66f212 WatchSource:0}: Error finding container 008eaec48c6be2df5076145f9b110c20db798e1cf64d18aabc701d826a66f212: Status 404 returned error can't find the container with id 008eaec48c6be2df5076145f9b110c20db798e1cf64d18aabc701d826a66f212
Jul 13 18:51:05.341898 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:05.341656    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0d6c1c_76f5_4da4_bbe3_2de78f4735f4.slice/crio-dba81c4781b8abc848ca99530d56403afe00e7c1aa724b30796879ae46155083 WatchSource:0}: Error finding container dba81c4781b8abc848ca99530d56403afe00e7c1aa724b30796879ae46155083: Status 404 returned error can't find the container with id dba81c4781b8abc848ca99530d56403afe00e7c1aa724b30796879ae46155083
Jul 13 18:51:06.279086 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.279057    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbdfd99cd-r9prl" event={"ID":"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4","Type":"ContainerStarted","Data":"dba81c4781b8abc848ca99530d56403afe00e7c1aa724b30796879ae46155083"}
Jul 13 18:51:06.280836 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.280806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqhbw" event={"ID":"54ed57eb-2265-4c86-9ca9-fbbaa753706f","Type":"ContainerStarted","Data":"e2b55f84cad65ad7a6e57533ad007693b67cf11dd5c4acb2d9a5f9305e0df691"}
Jul 13 18:51:06.283002 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.282979    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngb8f" event={"ID":"185e1d28-22e4-4b9f-bf9a-c49a8169e375","Type":"ContainerStarted","Data":"5babf345fe462769af64750ea667b695071a71e8dbf7227049caf71cce85f067"}
Jul 13 18:51:06.284264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.284239    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp" event={"ID":"83ae5ff5-1970-4f25-99f4-c53c7f19d710","Type":"ContainerStarted","Data":"008eaec48c6be2df5076145f9b110c20db798e1cf64d18aabc701d826a66f212"}
Jul 13 18:51:06.286298 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.286278    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tz6g6" event={"ID":"ec484b05-f937-4909-a863-02e7faa131f9","Type":"ContainerStarted","Data":"6d1fabc2d34d41f5db5a952868dbaa1f17f6244c812d7d5b211679a244a00ff2"}
Jul 13 18:51:06.286412 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.286320    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tz6g6" event={"ID":"ec484b05-f937-4909-a863-02e7faa131f9","Type":"ContainerStarted","Data":"452af87a03c26a8b566b438814839be3f6f8a03e12db0542d7a345901dca29f5"}
Jul 13 18:51:06.286412 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.286351    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:51:06.300484 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.300445    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pqhbw" podStartSLOduration=2.408986754 podStartE2EDuration="8.300429854s" totalImagesPullingTime="5.8914431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:06.299939932 +0000 UTC m=+58.405543551" watchObservedRunningTime="2026-07-13 18:51:06.300429854 +0000 UTC m=+58.406033457"
Jul 13 18:51:06.324864 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.324799    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tz6g6" podStartSLOduration=2.43074864 podStartE2EDuration="8.324777262s" totalImagesPullingTime="5.894028622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:06.324261506 +0000 UTC m=+58.429865128" watchObservedRunningTime="2026-07-13 18:51:06.324777262 +0000 UTC m=+58.430380881"
Jul 13 18:51:06.470903 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:06.470866    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:07.292878 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:07.292831    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngb8f" event={"ID":"185e1d28-22e4-4b9f-bf9a-c49a8169e375","Type":"ContainerStarted","Data":"2b00e59b3d5955821eaffa8b4296a1ddbfe972e415ae55178e257a81df9d3350"}
Jul 13 18:51:07.295111 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:07.295079    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp" event={"ID":"83ae5ff5-1970-4f25-99f4-c53c7f19d710","Type":"ContainerStarted","Data":"16457f3fb773fcf55ab695bc9c13555011791e51bdf73d54134da7722091cfa0"}
Jul 13 18:51:07.295250 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:07.295135    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp" event={"ID":"83ae5ff5-1970-4f25-99f4-c53c7f19d710","Type":"ContainerStarted","Data":"21954f37d0ac88f9c84a212d86c7fba415b605a0b7785ce0b8fe94b9766dffab"}
Jul 13 18:51:07.317542 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:07.317480    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-6vlsp" podStartSLOduration=4.891725457 podStartE2EDuration="6.317465027s" totalImagesPullingTime="1.42573957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:07.317435437 +0000 UTC m=+59.423039056" watchObservedRunningTime="2026-07-13 18:51:07.317465027 +0000 UTC m=+59.423068649"
Jul 13 18:51:07.778231 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:07.778194    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:07.869352 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:07.869315    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.125056 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:08.124945    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.207028 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:08.206984    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.548951 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:08.548916    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.663695 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:08.663663    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.754988 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:08.754960    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:08.990096 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:08.990063    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:09.028920 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.028885    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"]
Jul 13 18:51:09.032260 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.032240    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.035439 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.035421    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kf7mb\""
Jul 13 18:51:09.035643 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.035625    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 13 18:51:09.036597 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.036575    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 13 18:51:09.046724 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.046700    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"]
Jul 13 18:51:09.064673 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.064636    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"]
Jul 13 18:51:09.067291 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.067273    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.070264 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.070239    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 13 18:51:09.070542 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.070515    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 13 18:51:09.070883 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.070866    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5lr7f\""
Jul 13 18:51:09.070982 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.070928    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 13 18:51:09.076299 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.076281    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qm5c8"]
Jul 13 18:51:09.078800 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.078780    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.085808 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.085730    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 18:51:09.086066 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.086040    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 18:51:09.086277 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.086223    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bd5b6\""
Jul 13 18:51:09.086847 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.086664    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 18:51:09.091720 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.090051    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"]
Jul 13 18:51:09.098420 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098400    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvc9s\" (UniqueName: \"kubernetes.io/projected/72182912-d026-4984-acc5-3b9dc462c36b-kube-api-access-rvc9s\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.098516 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098437    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/72182912-d026-4984-acc5-3b9dc462c36b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.098516 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098460    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-tls\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.098516 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9112d63f-e06b-42ae-8399-a6e195c299af-sys\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.098516 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098504    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-textfile\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.098711 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9112d63f-e06b-42ae-8399-a6e195c299af-root\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.098711 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-wtmp\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.098711 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098662    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.098711 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9112d63f-e06b-42ae-8399-a6e195c299af-metrics-client-ca\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.098879 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098729    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs5g6\" (UniqueName: \"kubernetes.io/projected/4340d27b-b543-45f2-a746-eed158a97c28-kube-api-access-rs5g6\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.098879 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098792    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.098879 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098828    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72182912-d026-4984-acc5-3b9dc462c36b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.098978 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098928    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhp8z\" (UniqueName: \"kubernetes.io/projected/9112d63f-e06b-42ae-8399-a6e195c299af-kube-api-access-rhp8z\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.098978 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.099046 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.098991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.099084 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.099057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4340d27b-b543-45f2-a746-eed158a97c28-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.099127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.099078    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.099127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.099101    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4340d27b-b543-45f2-a746-eed158a97c28-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.099127 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.099119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4340d27b-b543-45f2-a746-eed158a97c28-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.104403 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.104383    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:09.200022 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.199981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvc9s\" (UniqueName: \"kubernetes.io/projected/72182912-d026-4984-acc5-3b9dc462c36b-kube-api-access-rvc9s\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.200201 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/72182912-d026-4984-acc5-3b9dc462c36b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.200201 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-tls\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200201 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200099    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9112d63f-e06b-42ae-8399-a6e195c299af-sys\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200201 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200124    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-textfile\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200201 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200150    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9112d63f-e06b-42ae-8399-a6e195c299af-root\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200201 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-wtmp\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200201 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200472 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:51:09.200206    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 13 18:51:09.200472 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:51:09.200275    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-tls podName:9112d63f-e06b-42ae-8399-a6e195c299af nodeName:}" failed. No retries permitted until 2026-07-13 18:51:09.700255214 +0000 UTC m=+61.805858828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-tls") pod "node-exporter-qm5c8" (UID: "9112d63f-e06b-42ae-8399-a6e195c299af") : secret "node-exporter-tls" not found
Jul 13 18:51:09.200472 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200268    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9112d63f-e06b-42ae-8399-a6e195c299af-sys\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200472 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200298    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9112d63f-e06b-42ae-8399-a6e195c299af-metrics-client-ca\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200472 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5g6\" (UniqueName: \"kubernetes.io/projected/4340d27b-b543-45f2-a746-eed158a97c28-kube-api-access-rs5g6\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.200472 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200376    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.200472 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9112d63f-e06b-42ae-8399-a6e195c299af-root\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72182912-d026-4984-acc5-3b9dc462c36b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.200773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200500    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-textfile\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhp8z\" (UniqueName: \"kubernetes.io/projected/9112d63f-e06b-42ae-8399-a6e195c299af-kube-api-access-rhp8z\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.200773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200630    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.200773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200664    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/72182912-d026-4984-acc5-3b9dc462c36b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.200773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200698    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4340d27b-b543-45f2-a746-eed158a97c28-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.200773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200728    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.200773 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4340d27b-b543-45f2-a746-eed158a97c28-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.201177 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.200788    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4340d27b-b543-45f2-a746-eed158a97c28-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.201177 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.201040    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9112d63f-e06b-42ae-8399-a6e195c299af-metrics-client-ca\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.201177 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.201168    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-wtmp\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.201294 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:51:09.201260    2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 13 18:51:09.201337 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:51:09.201306    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-tls podName:72182912-d026-4984-acc5-3b9dc462c36b nodeName:}" failed. No retries permitted until 2026-07-13 18:51:09.701291035 +0000 UTC m=+61.806894650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-5rqpt" (UID: "72182912-d026-4984-acc5-3b9dc462c36b") : secret "kube-state-metrics-tls" not found
Jul 13 18:51:09.201451 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.201430    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4340d27b-b543-45f2-a746-eed158a97c28-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.201739 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.201555    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72182912-d026-4984-acc5-3b9dc462c36b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.201834 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.201807    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-accelerators-collector-config\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.202034 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.201991    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.204808 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.204788    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.204928 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.204826    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4340d27b-b543-45f2-a746-eed158a97c28-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.204928 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.204863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4340d27b-b543-45f2-a746-eed158a97c28-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.204928 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.204891    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.211214 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.211192    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhp8z\" (UniqueName: \"kubernetes.io/projected/9112d63f-e06b-42ae-8399-a6e195c299af-kube-api-access-rhp8z\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.211451 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.211427    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5g6\" (UniqueName: \"kubernetes.io/projected/4340d27b-b543-45f2-a746-eed158a97c28-kube-api-access-rs5g6\") pod \"openshift-state-metrics-84bd68b896-9pdzz\" (UID: \"4340d27b-b543-45f2-a746-eed158a97c28\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.211569 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.211544    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvc9s\" (UniqueName: \"kubernetes.io/projected/72182912-d026-4984-acc5-3b9dc462c36b-kube-api-access-rvc9s\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.306580 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.306460    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:09.342502 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.342470    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"
Jul 13 18:51:09.415349 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.415314    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:09.530754 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.530720    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:09.705734 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.705694    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-tls\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.706133 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.705791    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.708417 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.708389    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9112d63f-e06b-42ae-8399-a6e195c299af-node-exporter-tls\") pod \"node-exporter-qm5c8\" (UID: \"9112d63f-e06b-42ae-8399-a6e195c299af\") " pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:09.708590 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.708574    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/72182912-d026-4984-acc5-3b9dc462c36b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-5rqpt\" (UID: \"72182912-d026-4984-acc5-3b9dc462c36b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.979306 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.979229    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:09.980451 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.980433    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5lr7f\""
Jul 13 18:51:09.988006 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.987977    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"
Jul 13 18:51:09.995258 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.995208    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bd5b6\""
Jul 13 18:51:09.997086 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.997066    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz"]
Jul 13 18:51:09.999737 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:09.999717    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qm5c8"
Jul 13 18:51:10.000468 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:10.000424    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4340d27b_b543_45f2_a746_eed158a97c28.slice/crio-1a0f9e0ff56f528aff9c2c90ba4c7c70f339004246608cb1f4bffd0f6e6fc4ff WatchSource:0}: Error finding container 1a0f9e0ff56f528aff9c2c90ba4c7c70f339004246608cb1f4bffd0f6e6fc4ff: Status 404 returned error can't find the container with id 1a0f9e0ff56f528aff9c2c90ba4c7c70f339004246608cb1f4bffd0f6e6fc4ff
Jul 13 18:51:10.012628 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:10.012598    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9112d63f_e06b_42ae_8399_a6e195c299af.slice/crio-02d6d04991752fa16dee9aeb608470d0073ca4302001fc2bdcc6f0f729bec5b2 WatchSource:0}: Error finding container 02d6d04991752fa16dee9aeb608470d0073ca4302001fc2bdcc6f0f729bec5b2: Status 404 returned error can't find the container with id 02d6d04991752fa16dee9aeb608470d0073ca4302001fc2bdcc6f0f729bec5b2
Jul 13 18:51:10.120452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.120406    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:10.122539 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.122501    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:51:10.129764 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.129734    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.133459 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.133440    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 13 18:51:10.133800 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.133781    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 13 18:51:10.133920 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.133800    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9b2vv\""
Jul 13 18:51:10.133992 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.133852    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 13 18:51:10.133992 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.133716    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 13 18:51:10.134097 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.133865    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 13 18:51:10.134406 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.134370    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 13 18:51:10.134510 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.134407    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 13 18:51:10.137456 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.137434    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 13 18:51:10.137590 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.137464    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 13 18:51:10.144484 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.144462    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-5rqpt"]
Jul 13 18:51:10.148313 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:10.148288    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72182912_d026_4984_acc5_3b9dc462c36b.slice/crio-351efa2612776e540015c4bd87824e792295164095ad32b040a0e043e60c3934 WatchSource:0}: Error finding container 351efa2612776e540015c4bd87824e792295164095ad32b040a0e043e60c3934: Status 404 returned error can't find the container with id 351efa2612776e540015c4bd87824e792295164095ad32b040a0e043e60c3934
Jul 13 18:51:10.151831 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.151799    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:51:10.203858 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.203832    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wftcq"
Jul 13 18:51:10.211970 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.211939    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212132 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.211995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-config-out\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212132 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212132 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212104    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212301 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212133    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212301 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212301 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212219    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-web-config\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212461 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-config-volume\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212461 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212369    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212461 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212397    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212461 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212436    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-tls-assets\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212681 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212468    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4bg\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-kube-api-access-np4bg\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.212681 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.212499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.263900 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.263839    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:10.307546 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.307501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngb8f" event={"ID":"185e1d28-22e4-4b9f-bf9a-c49a8169e375","Type":"ContainerStarted","Data":"275bd0889c61108c29b3dbbd68bb3e969f3d4154557a583b297d78bc1b46b6c2"}
Jul 13 18:51:10.308821 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.308781    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm5c8" event={"ID":"9112d63f-e06b-42ae-8399-a6e195c299af","Type":"ContainerStarted","Data":"02d6d04991752fa16dee9aeb608470d0073ca4302001fc2bdcc6f0f729bec5b2"}
Jul 13 18:51:10.310643 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.310396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbdfd99cd-r9prl" event={"ID":"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4","Type":"ContainerStarted","Data":"b8647dd639e7082b420ff30d2e18ddf5aa729603b2d9fd10fb36d3b64ecfe6e2"}
Jul 13 18:51:10.311959 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.311929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt" event={"ID":"72182912-d026-4984-acc5-3b9dc462c36b","Type":"ContainerStarted","Data":"351efa2612776e540015c4bd87824e792295164095ad32b040a0e043e60c3934"}
Jul 13 18:51:10.312848 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.312824    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.312968 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.312878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-web-config\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.312968 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.312911    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-config-volume\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.313081 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.312970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.313081 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.312999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.313081 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.313060    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-tls-assets\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.313223 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.313083    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4bg\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-kube-api-access-np4bg\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.313223 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.313115    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.313396 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.313371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.313647 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.313451    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-config-out\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.313647 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.313480    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.313647 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.313514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.315273 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.313868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.315273 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.314169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz" event={"ID":"4340d27b-b543-45f2-a746-eed158a97c28","Type":"ContainerStarted","Data":"029e8f972fae5fb56880eb31ddb239648dcd161298f6933a62e9dee9f94d99a5"}
Jul 13 18:51:10.315273 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.314223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz" event={"ID":"4340d27b-b543-45f2-a746-eed158a97c28","Type":"ContainerStarted","Data":"76f4c7d009b476a295560e1aec25c46a602c3cbcca6e64f735be0436cf945a3c"}
Jul 13 18:51:10.315273 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.314261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz" event={"ID":"4340d27b-b543-45f2-a746-eed158a97c28","Type":"ContainerStarted","Data":"1a0f9e0ff56f528aff9c2c90ba4c7c70f339004246608cb1f4bffd0f6e6fc4ff"}
Jul 13 18:51:10.315273 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.315174    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.315273 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.315200    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.316296 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.315858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.316296 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.316247    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-web-config\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.317620 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.317493    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.317620 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.317577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-tls-assets\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.317770 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.317697    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.317770 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.317763    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-config-out\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.318188 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.318148    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.318270 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.318197    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-config-volume\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.319287 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.319199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.320989 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.320961    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.325932 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.325904    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4bg\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-kube-api-access-np4bg\") pod \"alertmanager-main-0\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.343696 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.343660    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ngb8f" podStartSLOduration=1.9652735300000002 podStartE2EDuration="12.343646403s" totalImagesPullingTime="10.378372873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:10.341444463 +0000 UTC m=+62.447048082" watchObservedRunningTime="2026-07-13 18:51:10.343646403 +0000 UTC m=+62.449250014"
Jul 13 18:51:10.377926 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.377882    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bbdfd99cd-r9prl" podStartSLOduration=1.8775565950000002 podStartE2EDuration="6.377867012s" totalImagesPullingTime="4.500310417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:10.377425972 +0000 UTC m=+62.483029627" watchObservedRunningTime="2026-07-13 18:51:10.377867012 +0000 UTC m=+62.483470631"
Jul 13 18:51:10.397048 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.397021    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:10.439158 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.439127    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:10.524220 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.524153    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:10.610562 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.610435    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:51:10.613414 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:10.613387    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692933a6_f767_4df4_b6bb_7662eb17a867.slice/crio-25dcaa46269121151d4277a1cfe4d2c0158becca6dc80e7eb20f1eb22a9e80aa WatchSource:0}: Error finding container 25dcaa46269121151d4277a1cfe4d2c0158becca6dc80e7eb20f1eb22a9e80aa: Status 404 returned error can't find the container with id 25dcaa46269121151d4277a1cfe4d2c0158becca6dc80e7eb20f1eb22a9e80aa
Jul 13 18:51:10.628117 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:10.628091    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:11.320341 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:11.320296    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerStarted","Data":"25dcaa46269121151d4277a1cfe4d2c0158becca6dc80e7eb20f1eb22a9e80aa"}
Jul 13 18:51:11.510914 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:11.510854    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:11.617128 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:11.617048    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.019318 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:12.019281    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.097542 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:12.097507    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.198564 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:12.198513    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.298514 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:12.298430    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:12.400564 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:12.400524    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:13.327822 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.327783    2569 generic.go:356] "Generic (PLEG): container finished" podID="9112d63f-e06b-42ae-8399-a6e195c299af" containerID="c62b6e4234807fe2bcccb1cf11fb40fce8dbc27aa3e9c50d1b966ab6e173ae40" exitCode=0
Jul 13 18:51:13.328007 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.327870    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm5c8" event={"ID":"9112d63f-e06b-42ae-8399-a6e195c299af","Type":"ContainerDied","Data":"c62b6e4234807fe2bcccb1cf11fb40fce8dbc27aa3e9c50d1b966ab6e173ae40"}
Jul 13 18:51:13.330043 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.330020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt" event={"ID":"72182912-d026-4984-acc5-3b9dc462c36b","Type":"ContainerStarted","Data":"76668a9b9ccb39f7059d90465dcf2713f33ff94e7473576395dfc22a7895d934"}
Jul 13 18:51:13.330166 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.330081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt" event={"ID":"72182912-d026-4984-acc5-3b9dc462c36b","Type":"ContainerStarted","Data":"1b7a384b70fa7839be04e55f719faa39c3ab0cbb501447e79ad73e4e653e50d4"}
Jul 13 18:51:13.330166 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.330110    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt" event={"ID":"72182912-d026-4984-acc5-3b9dc462c36b","Type":"ContainerStarted","Data":"96edeb95e5556caf6d874a3141d0d943bb208ddaba46c44185dc03d5de7f8052"}
Jul 13 18:51:13.332325 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.332302    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz" event={"ID":"4340d27b-b543-45f2-a746-eed158a97c28","Type":"ContainerStarted","Data":"387fba08f4d77e12c285b0b16ffdc5e4e6c44453fc9130aee67c50547b10154e"}
Jul 13 18:51:13.333859 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.333836    2569 generic.go:356] "Generic (PLEG): container finished" podID="692933a6-f767-4df4-b6bb-7662eb17a867" containerID="66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487" exitCode=0
Jul 13 18:51:13.333961 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.333864    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerDied","Data":"66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487"}
Jul 13 18:51:13.453206 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.453152    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-9pdzz" podStartSLOduration=2.253583081 podStartE2EDuration="4.453131683s" totalImagesPullingTime="2.199548602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:13.427901004 +0000 UTC m=+65.533504623" watchObservedRunningTime="2026-07-13 18:51:13.453131683 +0000 UTC m=+65.558735303"
Jul 13 18:51:13.453676 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.453640    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-5rqpt" podStartSLOduration=2.25201405 podStartE2EDuration="4.453627723s" totalImagesPullingTime="2.201613673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:13.452098014 +0000 UTC m=+65.557701645" watchObservedRunningTime="2026-07-13 18:51:13.453627723 +0000 UTC m=+65.559231344"
Jul 13 18:51:13.731827 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.731791    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:13.830959 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.830919    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:13.917924 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.917889    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbdfd99cd-r9prl"]
Jul 13 18:51:13.953954 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.953916    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-599d97cf86-6vrtz"]
Jul 13 18:51:13.956106 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.956085    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:13.964008 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.963985    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 13 18:51:13.968653 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:13.968632    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599d97cf86-6vrtz"]
Jul 13 18:51:14.034539 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.034450    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:14.046282 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.046253    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-oauth-config\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.046423 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.046300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-trusted-ca-bundle\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.046423 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.046316    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxmn\" (UniqueName: \"kubernetes.io/projected/93947031-85b0-458d-ac87-ba1107954798-kube-api-access-ctxmn\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.046423 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.046340    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-service-ca\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.046423 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.046373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-console-config\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.046423 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.046396    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-oauth-serving-cert\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.046614 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.046444    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-serving-cert\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.147474 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.147443    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-oauth-config\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.147655 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.147485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-trusted-ca-bundle\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.147655 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.147505    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxmn\" (UniqueName: \"kubernetes.io/projected/93947031-85b0-458d-ac87-ba1107954798-kube-api-access-ctxmn\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.147655 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.147549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-service-ca\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.147655 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.147575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-console-config\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.147655 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.147595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-oauth-serving-cert\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.147655 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.147645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-serving-cert\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.148392 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.148335    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-console-config\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.148392 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.148372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-service-ca\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.148553 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.148449    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-oauth-serving-cert\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.148553 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.148485    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-trusted-ca-bundle\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.149914 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.149892    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-oauth-config\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.150195 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.150175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-serving-cert\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.155607 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.155585    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxmn\" (UniqueName: \"kubernetes.io/projected/93947031-85b0-458d-ac87-ba1107954798-kube-api-access-ctxmn\") pod \"console-599d97cf86-6vrtz\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") " pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.200098 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.200066    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5db645677c-8jj4h"]
Jul 13 18:51:14.203206 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.203187    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.206047 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.206024    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bfb5x\""
Jul 13 18:51:14.206134 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.206075    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 13 18:51:14.206134 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.206103    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 13 18:51:14.206348 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.206328    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 13 18:51:14.206442 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.206397    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 13 18:51:14.206587 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.206572    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 13 18:51:14.212591 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.212570    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 13 18:51:14.218169 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.218149    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5db645677c-8jj4h"]
Jul 13 18:51:14.248478 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.248436    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb17b840-15ee-4f40-9a16-264becb9700e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.248650 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.248502    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-telemeter-client-tls\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.248650 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.248569    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb17b840-15ee-4f40-9a16-264becb9700e-metrics-client-ca\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.248650 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.248613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.248814 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.248652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-federate-client-tls\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.248814 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.248696    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-secret-telemeter-client\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.248814 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.248732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb17b840-15ee-4f40-9a16-264becb9700e-serving-certs-ca-bundle\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.248814 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.248755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpmtb\" (UniqueName: \"kubernetes.io/projected/bb17b840-15ee-4f40-9a16-264becb9700e-kube-api-access-gpmtb\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.265449 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.265428    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:14.296541 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.296463    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:14.341044 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.341007    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm5c8" event={"ID":"9112d63f-e06b-42ae-8399-a6e195c299af","Type":"ContainerStarted","Data":"b8000ef4b48ad3a57bb291dbb24937d98bc8d41adbd8e5e2884e5218ebff25db"}
Jul 13 18:51:14.341133 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.341080    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qm5c8" event={"ID":"9112d63f-e06b-42ae-8399-a6e195c299af","Type":"ContainerStarted","Data":"f0a54573eb84f365babf84d09dfed52abfbd2e4fba4d6156208360f76b6141f5"}
Jul 13 18:51:14.349260 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.349234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.349391 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.349272    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-federate-client-tls\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.349391 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.349304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-secret-telemeter-client\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.349391 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.349334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb17b840-15ee-4f40-9a16-264becb9700e-serving-certs-ca-bundle\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.349391 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.349361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmtb\" (UniqueName: \"kubernetes.io/projected/bb17b840-15ee-4f40-9a16-264becb9700e-kube-api-access-gpmtb\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.349612 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.349430    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb17b840-15ee-4f40-9a16-264becb9700e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.349612 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.349469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-telemeter-client-tls\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.349612 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.349498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb17b840-15ee-4f40-9a16-264becb9700e-metrics-client-ca\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.350453 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.350216    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb17b840-15ee-4f40-9a16-264becb9700e-serving-certs-ca-bundle\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.350453 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.350322    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb17b840-15ee-4f40-9a16-264becb9700e-metrics-client-ca\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.351049 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.351003    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb17b840-15ee-4f40-9a16-264becb9700e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.352002 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.351977    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-secret-telemeter-client\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.352290 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.352272    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.352523 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.352506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-telemeter-client-tls\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.352746 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.352729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bb17b840-15ee-4f40-9a16-264becb9700e-federate-client-tls\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.358846 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.358802    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmtb\" (UniqueName: \"kubernetes.io/projected/bb17b840-15ee-4f40-9a16-264becb9700e-kube-api-access-gpmtb\") pod \"telemeter-client-5db645677c-8jj4h\" (UID: \"bb17b840-15ee-4f40-9a16-264becb9700e\") " pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.363550 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.363468    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qm5c8" podStartSLOduration=2.947031521 podStartE2EDuration="5.36345024s" totalImagesPullingTime="2.334367758s" totalInitContainerRuntime="82.050961ms" isStatefulPod=true podCreationTimestamp="2026-07-13 18:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:14.361422687 +0000 UTC m=+66.467026298" watchObservedRunningTime="2026-07-13 18:51:14.36345024 +0000 UTC m=+66.469053860"
Jul 13 18:51:14.364367 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.364344    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:14.399000 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:14.396282    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93947031_85b0_458d_ac87_ba1107954798.slice/crio-a696c5c5e616662f761ad1e6bc6e47e7ac66977749755395c136e5770d2b6b1a WatchSource:0}: Error finding container a696c5c5e616662f761ad1e6bc6e47e7ac66977749755395c136e5770d2b6b1a: Status 404 returned error can't find the container with id a696c5c5e616662f761ad1e6bc6e47e7ac66977749755395c136e5770d2b6b1a
Jul 13 18:51:14.399000 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.398397    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-599d97cf86-6vrtz"]
Jul 13 18:51:14.513825 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.513794    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h"
Jul 13 18:51:14.638646 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.638619    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5db645677c-8jj4h"]
Jul 13 18:51:14.640617 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:14.640584    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb17b840_15ee_4f40_9a16_264becb9700e.slice/crio-251aeaf27b0bf1ab14f55fb9b29420a0ca9a7faf21c2a2381088ce62d09e8963 WatchSource:0}: Error finding container 251aeaf27b0bf1ab14f55fb9b29420a0ca9a7faf21c2a2381088ce62d09e8963: Status 404 returned error can't find the container with id 251aeaf27b0bf1ab14f55fb9b29420a0ca9a7faf21c2a2381088ce62d09e8963
Jul 13 18:51:14.674385 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:14.674356    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:15.229165 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.229122    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:51:15.230939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.230908    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:15.232488 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.232465    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.236280 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.236142    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 13 18:51:15.236556 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.236442    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 13 18:51:15.236556 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.236477    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eef1nuotvtqv\""
Jul 13 18:51:15.236759 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.236565    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 13 18:51:15.236823 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.236766    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 13 18:51:15.236823 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.236801    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 13 18:51:15.237521 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.237400    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 13 18:51:15.237676 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.237662    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 13 18:51:15.237760 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.237671    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 13 18:51:15.237909 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.237893    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 13 18:51:15.239073 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.238752    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 13 18:51:15.239073 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.238778    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hqf4n\""
Jul 13 18:51:15.239073 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.238895    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 13 18:51:15.239073 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.238978    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 13 18:51:15.239650 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.239631    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 13 18:51:15.239886 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.239823    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 13 18:51:15.240249 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.240229    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 13 18:51:15.252598 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.252574    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:51:15.348186 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.348151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h" event={"ID":"bb17b840-15ee-4f40-9a16-264becb9700e","Type":"ContainerStarted","Data":"251aeaf27b0bf1ab14f55fb9b29420a0ca9a7faf21c2a2381088ce62d09e8963"}
Jul 13 18:51:15.350134 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.350100    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599d97cf86-6vrtz" event={"ID":"93947031-85b0-458d-ac87-ba1107954798","Type":"ContainerStarted","Data":"3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0"}
Jul 13 18:51:15.350277 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.350147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599d97cf86-6vrtz" event={"ID":"93947031-85b0-458d-ac87-ba1107954798","Type":"ContainerStarted","Data":"a696c5c5e616662f761ad1e6bc6e47e7ac66977749755395c136e5770d2b6b1a"}
Jul 13 18:51:15.361434 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config-out\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361600 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwps4\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-kube-api-access-cwps4\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361600 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361478    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-web-config\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361600 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361570    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361769 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361769 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361636    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361769 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361769 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361734    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361904 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361792    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361904 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361981 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361896    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361981 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361949    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.361981 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361973    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.362121 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.361988    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.362121 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.362029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.362121 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.362057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.362121 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.362118    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.362321 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.362146    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.362321 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.362283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.362423 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.362321    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.371393 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.371325    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-599d97cf86-6vrtz" podStartSLOduration=2.371304188 podStartE2EDuration="2.371304188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:15.371195196 +0000 UTC m=+67.476798816" watchObservedRunningTime="2026-07-13 18:51:15.371304188 +0000 UTC m=+67.476907808"
Jul 13 18:51:15.463835 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.463791    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.464027 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.463849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.464027 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.463877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.464597 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.464409    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.464597 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.464448    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.464597 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.464484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.464597 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.464506    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.464881 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.464608    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.464881 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.464648    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.464985 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.464944    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465042 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.464985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465042 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465142 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465039    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465142 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465239 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465239 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465239 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465388 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465254    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465388 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465329    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465388 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465362    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config-out\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465388 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465388 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465386    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwps4\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-kube-api-access-cwps4\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.465665 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-web-config\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.466023 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.465996    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.466703 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.466415    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.466703 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.466444    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.469336 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.469310    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.470083 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.470061    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.470399 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.470355    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.470984 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.470937    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.471389 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.471365    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.471806 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.471782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-web-config\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.475588 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.472019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.475588 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.474856    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.475588 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.474857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.475588 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.474911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.476369 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.476202    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config-out\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.476742 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.476689    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.477251 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.477226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwps4\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-kube-api-access-cwps4\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.478711 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.478673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.545805 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.545468    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:15.670037 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.669797    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:51:15.673376 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.673160    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 18:51:15.684991 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.684778    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 18:51:15.695610 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.695552    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvfr\" (UniqueName: \"kubernetes.io/projected/8d506c0b-8324-498e-962e-6e396a66159e-kube-api-access-dzvfr\") pod \"network-check-target-5kc4j\" (UID: \"8d506c0b-8324-498e-962e-6e396a66159e\") " pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:51:15.734897 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.734613    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:51:15.737032 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:15.737005    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf545e0_3407_41c2_beb0_d0f6e1e89eb0.slice/crio-727c3f7f4c97e13cc142e5c5daeb9107c7f3abb44a408fe95f554888ef591c21 WatchSource:0}: Error finding container 727c3f7f4c97e13cc142e5c5daeb9107c7f3abb44a408fe95f554888ef591c21: Status 404 returned error can't find the container with id 727c3f7f4c97e13cc142e5c5daeb9107c7f3abb44a408fe95f554888ef591c21
Jul 13 18:51:15.770698 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.770667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:51:15.770848 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.770826    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:51:15.773591 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.773568    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:51:15.773708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.773639    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:51:15.784058 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.784022    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03610f49-0413-408e-9e5c-d1adc6d46204-original-pull-secret\") pod \"global-pull-secret-syncer-pg7df\" (UID: \"03610f49-0413-408e-9e5c-d1adc6d46204\") " pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:51:15.784668 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.784647    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84-metrics-certs\") pod \"network-metrics-daemon-pdlbh\" (UID: \"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84\") " pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:51:15.854351 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.854318    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nk8z7\""
Jul 13 18:51:15.855617 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.855401    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg7df"
Jul 13 18:51:15.861447 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.861419    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:51:15.961298 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.961226    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfqjl\""
Jul 13 18:51:15.969114 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:15.968905    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlbh"
Jul 13 18:51:16.054666 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.054636    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pg7df"]
Jul 13 18:51:16.066911 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.066889    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5kc4j"]
Jul 13 18:51:16.201763 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.201732    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:16.283781 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.283749    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:16.298771 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.298670    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tz6g6"
Jul 13 18:51:16.331980 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:16.331902    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03610f49_0413_408e_9e5c_d1adc6d46204.slice/crio-326c93dc35cf9b2528c90f3e35f5f2fca63814b817dd755e3921cdcbee0ac348 WatchSource:0}: Error finding container 326c93dc35cf9b2528c90f3e35f5f2fca63814b817dd755e3921cdcbee0ac348: Status 404 returned error can't find the container with id 326c93dc35cf9b2528c90f3e35f5f2fca63814b817dd755e3921cdcbee0ac348
Jul 13 18:51:16.332566 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:16.332539    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d506c0b_8324_498e_962e_6e396a66159e.slice/crio-bd165459bb7c5a259b434d4602d90df796216b980877214ca2e139d3987bd04a WatchSource:0}: Error finding container bd165459bb7c5a259b434d4602d90df796216b980877214ca2e139d3987bd04a: Status 404 returned error can't find the container with id bd165459bb7c5a259b434d4602d90df796216b980877214ca2e139d3987bd04a
Jul 13 18:51:16.356111 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.356065    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5kc4j" event={"ID":"8d506c0b-8324-498e-962e-6e396a66159e","Type":"ContainerStarted","Data":"bd165459bb7c5a259b434d4602d90df796216b980877214ca2e139d3987bd04a"}
Jul 13 18:51:16.358551 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.358503    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerID="1356fdcdb21fc801d12e14512db5f42bc64a7a320412231c227deae845e06590" exitCode=0
Jul 13 18:51:16.358731 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.358593    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerDied","Data":"1356fdcdb21fc801d12e14512db5f42bc64a7a320412231c227deae845e06590"}
Jul 13 18:51:16.358731 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.358679    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerStarted","Data":"727c3f7f4c97e13cc142e5c5daeb9107c7f3abb44a408fe95f554888ef591c21"}
Jul 13 18:51:16.359933 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.359907    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pg7df" event={"ID":"03610f49-0413-408e-9e5c-d1adc6d46204","Type":"ContainerStarted","Data":"326c93dc35cf9b2528c90f3e35f5f2fca63814b817dd755e3921cdcbee0ac348"}
Jul 13 18:51:16.366286 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.366263    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:16.469012 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.468979    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdlbh"]
Jul 13 18:51:16.470229 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:16.470201    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:16.878442 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:16.878403    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc297bb5d_97f3_4da7_8eb0_eb0bab9e5c84.slice/crio-4081ffb18a5972c5843f1f39549c86e180decaf4f831c3c9c3dd9dd6368f9b9c WatchSource:0}: Error finding container 4081ffb18a5972c5843f1f39549c86e180decaf4f831c3c9c3dd9dd6368f9b9c: Status 404 returned error can't find the container with id 4081ffb18a5972c5843f1f39549c86e180decaf4f831c3c9c3dd9dd6368f9b9c
Jul 13 18:51:17.374096 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.374056    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerStarted","Data":"ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed"}
Jul 13 18:51:17.374263 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.374150    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerStarted","Data":"adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1"}
Jul 13 18:51:17.374263 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.374194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerStarted","Data":"10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739"}
Jul 13 18:51:17.374263 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.374231    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerStarted","Data":"eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb"}
Jul 13 18:51:17.374424 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.374271    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerStarted","Data":"d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1"}
Jul 13 18:51:17.378300 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.377251    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h" event={"ID":"bb17b840-15ee-4f40-9a16-264becb9700e","Type":"ContainerStarted","Data":"61f587d7e25b1433f2a3d4096883fdf3efc6d15878fde58e5085aa2a10e6f14a"}
Jul 13 18:51:17.378300 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.377310    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h" event={"ID":"bb17b840-15ee-4f40-9a16-264becb9700e","Type":"ContainerStarted","Data":"a08fa67fc8a1043a088be89a10c5268a6249913de1d042bc540a9f18e8f215f2"}
Jul 13 18:51:17.378300 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.377342    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h" event={"ID":"bb17b840-15ee-4f40-9a16-264becb9700e","Type":"ContainerStarted","Data":"be3b883fa1eb4e8683dff864719e60a23a113eeb639f074f31ef116151cb097f"}
Jul 13 18:51:17.380962 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.380431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdlbh" event={"ID":"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84","Type":"ContainerStarted","Data":"4081ffb18a5972c5843f1f39549c86e180decaf4f831c3c9c3dd9dd6368f9b9c"}
Jul 13 18:51:17.407572 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:17.407485    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5db645677c-8jj4h" podStartSLOduration=1.141162634 podStartE2EDuration="3.407450607s" totalImagesPullingTime="2.266287973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:17.406168172 +0000 UTC m=+69.511771792" watchObservedRunningTime="2026-07-13 18:51:17.407450607 +0000 UTC m=+69.513054227"
Jul 13 18:51:18.123659 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.123094    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c4fd9f694-zb6lp"]
Jul 13 18:51:18.140599 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.140348    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.148556 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.148205    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c4fd9f694-zb6lp"]
Jul 13 18:51:18.193720 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.193682    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-config\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.193889 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.193735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-oauth-config\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.193889 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.193767    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-oauth-serving-cert\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.193889 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.193801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-service-ca\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.193889 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.193866    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-serving-cert\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.194122 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.193893    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-trusted-ca-bundle\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.194122 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.193921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5cmf\" (UniqueName: \"kubernetes.io/projected/d12a3724-c4f4-4146-8c0e-2ef9d9130220-kube-api-access-z5cmf\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.209974 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.209879    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:18.272267 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.272126    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:18.295702 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.295061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-trusted-ca-bundle\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.295702 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.295106    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5cmf\" (UniqueName: \"kubernetes.io/projected/d12a3724-c4f4-4146-8c0e-2ef9d9130220-kube-api-access-z5cmf\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.295702 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.295194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-config\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.295702 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.295227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-oauth-config\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.295702 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.295251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-oauth-serving-cert\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.295702 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.295282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-service-ca\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.295702 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.295338    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-serving-cert\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.296352 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.296323    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-config\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.296647 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.296619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-service-ca\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.297189 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.297146    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-trusted-ca-bundle\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.298034 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.297992    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-oauth-serving-cert\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.298619 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.298566    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-serving-cert\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.299169 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.299081    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-oauth-config\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.312249 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.312215    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5cmf\" (UniqueName: \"kubernetes.io/projected/d12a3724-c4f4-4146-8c0e-2ef9d9130220-kube-api-access-z5cmf\") pod \"console-5c4fd9f694-zb6lp\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") " pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:18.460688 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:18.460653    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:19.013658 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.013627    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c4fd9f694-zb6lp"]
Jul 13 18:51:19.018021 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:51:19.017982    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12a3724_c4f4_4146_8c0e_2ef9d9130220.slice/crio-939102c8ef57a0bd078c9e5b003ee98b6515e1ace7341835c089a577f3d75661 WatchSource:0}: Error finding container 939102c8ef57a0bd078c9e5b003ee98b6515e1ace7341835c089a577f3d75661: Status 404 returned error can't find the container with id 939102c8ef57a0bd078c9e5b003ee98b6515e1ace7341835c089a577f3d75661
Jul 13 18:51:19.399434 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.398805    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4fd9f694-zb6lp" event={"ID":"d12a3724-c4f4-4146-8c0e-2ef9d9130220","Type":"ContainerStarted","Data":"9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440"}
Jul 13 18:51:19.399434 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.398854    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4fd9f694-zb6lp" event={"ID":"d12a3724-c4f4-4146-8c0e-2ef9d9130220","Type":"ContainerStarted","Data":"939102c8ef57a0bd078c9e5b003ee98b6515e1ace7341835c089a577f3d75661"}
Jul 13 18:51:19.404207 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.404175    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerStarted","Data":"d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd"}
Jul 13 18:51:19.406496 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.406421    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdlbh" event={"ID":"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84","Type":"ContainerStarted","Data":"64c86fd81c4134d82ace44bc3e25d1d4c7a23a8faea51e00af736eb8a6a0b01a"}
Jul 13 18:51:19.406496 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.406482    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdlbh" event={"ID":"c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84","Type":"ContainerStarted","Data":"bfa58a0246a8fa95c5fb771a4d4cc944350517eaddc642fb557af8a48e84213e"}
Jul 13 18:51:19.426388 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.426242    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c4fd9f694-zb6lp" podStartSLOduration=1.426225336 podStartE2EDuration="1.426225336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:19.425282738 +0000 UTC m=+71.530886357" watchObservedRunningTime="2026-07-13 18:51:19.426225336 +0000 UTC m=+71.531828956"
Jul 13 18:51:19.454639 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.454577    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pdlbh" podStartSLOduration=67.49287895 podStartE2EDuration="1m9.454554844s" totalImagesPullingTime="1.961675894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:19.453721751 +0000 UTC m=+71.559325371" watchObservedRunningTime="2026-07-13 18:51:19.454554844 +0000 UTC m=+71.560158464"
Jul 13 18:51:19.500442 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.500243    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.998353291 podStartE2EDuration="9.500121962s" totalImagesPullingTime="6.477696589s" totalInitContainerRuntime="24.072082ms" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:19.495585926 +0000 UTC m=+71.601189546" watchObservedRunningTime="2026-07-13 18:51:19.500121962 +0000 UTC m=+71.605725588"
Jul 13 18:51:19.693269 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.693146    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:19.795277 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:19.795236    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:20.684988 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:20.684954    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:20.783898 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:20.783810    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:22.226244 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:22.226214    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:22.328122 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:22.328092    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:23.428552 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:23.428452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5kc4j" event={"ID":"8d506c0b-8324-498e-962e-6e396a66159e","Type":"ContainerStarted","Data":"1ef34799a808c922374ea02bdaba97e56b7dd178ec12d4fceabc25b053395886"}
Jul 13 18:51:23.428925 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:23.428778    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:51:23.431411 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:23.431387    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerStarted","Data":"82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce"}
Jul 13 18:51:23.435974 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:23.435948    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pg7df" event={"ID":"03610f49-0413-408e-9e5c-d1adc6d46204","Type":"ContainerStarted","Data":"53ba784654c10d06192e5f8412393466f30002e7697ad3b95b60045c8eec74f8"}
Jul 13 18:51:23.453406 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:23.453352    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5kc4j" podStartSLOduration=66.514298959 podStartE2EDuration="1m13.453331271s" totalImagesPullingTime="6.939032312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:23.452770897 +0000 UTC m=+75.558374516" watchObservedRunningTime="2026-07-13 18:51:23.453331271 +0000 UTC m=+75.558934883"
Jul 13 18:51:24.265820 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:24.265786    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:24.265992 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:24.265844    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:24.270519 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:24.270495    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:24.298706 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:24.298654    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pg7df" podStartSLOduration=67.348559514 podStartE2EDuration="1m14.298637603s" totalImagesPullingTime="6.950078089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:23.479357933 +0000 UTC m=+75.584961552" watchObservedRunningTime="2026-07-13 18:51:24.298637603 +0000 UTC m=+76.404241221"
Jul 13 18:51:24.444094 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:24.444055    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerStarted","Data":"f8aa9f85452d26368f6636c21b9cab2511c95fac77b243c65abbb30d90227fce"}
Jul 13 18:51:24.448613 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:24.448590    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:24.574153 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:24.574070    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:24.693113 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:24.693077    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:25.373035 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:25.372905    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:25.494685 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:25.494642    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:26.454820 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:26.454770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerStarted","Data":"29274eab4d8dd470b41ee1c2115afa388aa498a890ff2489db00419c886a4eff"}
Jul 13 18:51:26.455017 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:26.454848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerStarted","Data":"465b67ac295fa53c7d82a20591b546e6e10e6fd710c27afd620a56276917bdba"}
Jul 13 18:51:26.455017 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:26.454890    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerStarted","Data":"03d29e0d18cabac1b0eb5e765781bdd1be3480d742f6397525808d76c3ccdd6a"}
Jul 13 18:51:26.455017 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:26.454919    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerStarted","Data":"7e09683d89a4c2a8ca4c9f98933e4a0447f048cd1dd603cbf01157dd29c641c5"}
Jul 13 18:51:26.490884 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:26.490824    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.568080647 podStartE2EDuration="11.490808304s" totalImagesPullingTime="8.895940321s" totalInitContainerRuntime="26.787336ms" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:26.48815736 +0000 UTC m=+78.593760978" watchObservedRunningTime="2026-07-13 18:51:26.490808304 +0000 UTC m=+78.596411922"
Jul 13 18:51:27.176447 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:27.176409    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:27.493844 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:27.493770    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:27.600839 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:27.600808    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:28.460857 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:28.460828    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:28.461304 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:28.460884    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:28.465771 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:28.465747    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:29.470806 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:29.470768    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:51:29.552170 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:29.552134    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599d97cf86-6vrtz"]
Jul 13 18:51:29.859565 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:29.859465    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:30.007978 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:30.007942    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:30.546056 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:30.546029    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:51:32.348159 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:32.348128    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:32.442890 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:32.442856    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:38.937823 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:38.937780    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bbdfd99cd-r9prl" podUID="ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" containerName="console" containerID="cri-o://b8647dd639e7082b420ff30d2e18ddf5aa729603b2d9fd10fb36d3b64ecfe6e2" gracePeriod=15
Jul 13 18:51:39.508878 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.508841    2569 generic.go:356] "Generic (PLEG): container finished" podID="ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" containerID="b8647dd639e7082b420ff30d2e18ddf5aa729603b2d9fd10fb36d3b64ecfe6e2" exitCode=0
Jul 13 18:51:39.509061 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.508906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbdfd99cd-r9prl" event={"ID":"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4","Type":"ContainerDied","Data":"b8647dd639e7082b420ff30d2e18ddf5aa729603b2d9fd10fb36d3b64ecfe6e2"}
Jul 13 18:51:39.735412 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.735380    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:39.800035 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.799956    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-oauth-config\") pod \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") "
Jul 13 18:51:39.800035 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.799996    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-config\") pod \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") "
Jul 13 18:51:39.800035 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.800015    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-serving-cert\") pod \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") "
Jul 13 18:51:39.800249 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.800049    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvhp2\" (UniqueName: \"kubernetes.io/projected/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-kube-api-access-rvhp2\") pod \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") "
Jul 13 18:51:39.800249 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.800183    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-service-ca\") pod \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") "
Jul 13 18:51:39.800326 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.800254    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-oauth-serving-cert\") pod \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\" (UID: \"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4\") "
Jul 13 18:51:39.800483 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.800451    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-config" (OuterVolumeSpecName: "console-config") pod "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" (UID: "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:39.800631 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.800570    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-service-ca" (OuterVolumeSpecName: "service-ca") pod "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" (UID: "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:39.800695 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.800672    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" (UID: "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:39.802446 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.802413    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" (UID: "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:51:39.802565 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.802458    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" (UID: "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:51:39.802565 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.802465    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-kube-api-access-rvhp2" (OuterVolumeSpecName: "kube-api-access-rvhp2") pod "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" (UID: "ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4"). InnerVolumeSpecName "kube-api-access-rvhp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:51:39.901240 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.901204    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-oauth-serving-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:39.901240 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.901234    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-oauth-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:39.901240 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.901244    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:39.901462 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.901253    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-console-serving-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:39.901462 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.901262    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvhp2\" (UniqueName: \"kubernetes.io/projected/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-kube-api-access-rvhp2\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:39.901462 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:39.901271    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4-service-ca\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:40.364140 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:40.364057    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:40.483468 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:40.483438    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:40.514100 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:40.514075    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbdfd99cd-r9prl"
Jul 13 18:51:40.514275 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:40.514075    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbdfd99cd-r9prl" event={"ID":"ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4","Type":"ContainerDied","Data":"dba81c4781b8abc848ca99530d56403afe00e7c1aa724b30796879ae46155083"}
Jul 13 18:51:40.514275 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:40.514209    2569 scope.go:117] "RemoveContainer" containerID="b8647dd639e7082b420ff30d2e18ddf5aa729603b2d9fd10fb36d3b64ecfe6e2"
Jul 13 18:51:40.537551 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:40.537515    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbdfd99cd-r9prl"]
Jul 13 18:51:40.542701 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:40.542678    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bbdfd99cd-r9prl"]
Jul 13 18:51:42.044832 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:42.044801    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" path="/var/lib/kubelet/pods/ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4/volumes"
Jul 13 18:51:48.056295 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:48.056263    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:48.543580 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:48.543544    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:48.623879 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:48.623849    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:51:54.448101 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.448065    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5kc4j"
Jul 13 18:51:54.571590 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.571513    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-599d97cf86-6vrtz" podUID="93947031-85b0-458d-ac87-ba1107954798" containerName="console" containerID="cri-o://3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0" gracePeriod=15
Jul 13 18:51:54.868055 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.868026    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:54.927965 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.927938    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-console-config\") pod \"93947031-85b0-458d-ac87-ba1107954798\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") "
Jul 13 18:51:54.928148 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.927972    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-trusted-ca-bundle\") pod \"93947031-85b0-458d-ac87-ba1107954798\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") "
Jul 13 18:51:54.928148 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.927993    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-oauth-config\") pod \"93947031-85b0-458d-ac87-ba1107954798\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") "
Jul 13 18:51:54.928148 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928025    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-oauth-serving-cert\") pod \"93947031-85b0-458d-ac87-ba1107954798\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") "
Jul 13 18:51:54.928148 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928049    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-serving-cert\") pod \"93947031-85b0-458d-ac87-ba1107954798\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") "
Jul 13 18:51:54.928148 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928105    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-service-ca\") pod \"93947031-85b0-458d-ac87-ba1107954798\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") "
Jul 13 18:51:54.928148 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928149    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctxmn\" (UniqueName: \"kubernetes.io/projected/93947031-85b0-458d-ac87-ba1107954798-kube-api-access-ctxmn\") pod \"93947031-85b0-458d-ac87-ba1107954798\" (UID: \"93947031-85b0-458d-ac87-ba1107954798\") "
Jul 13 18:51:54.928451 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928321    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-console-config" (OuterVolumeSpecName: "console-config") pod "93947031-85b0-458d-ac87-ba1107954798" (UID: "93947031-85b0-458d-ac87-ba1107954798"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:54.928451 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928362    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "93947031-85b0-458d-ac87-ba1107954798" (UID: "93947031-85b0-458d-ac87-ba1107954798"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:54.928566 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928482    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-console-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:54.928566 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928501    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-trusted-ca-bundle\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:54.928716 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928683    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "93947031-85b0-458d-ac87-ba1107954798" (UID: "93947031-85b0-458d-ac87-ba1107954798"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:54.928803 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.928734    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-service-ca" (OuterVolumeSpecName: "service-ca") pod "93947031-85b0-458d-ac87-ba1107954798" (UID: "93947031-85b0-458d-ac87-ba1107954798"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:51:54.930595 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.930569    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93947031-85b0-458d-ac87-ba1107954798-kube-api-access-ctxmn" (OuterVolumeSpecName: "kube-api-access-ctxmn") pod "93947031-85b0-458d-ac87-ba1107954798" (UID: "93947031-85b0-458d-ac87-ba1107954798"). InnerVolumeSpecName "kube-api-access-ctxmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:51:54.930595 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.930581    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "93947031-85b0-458d-ac87-ba1107954798" (UID: "93947031-85b0-458d-ac87-ba1107954798"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:51:54.930736 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:54.930685    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "93947031-85b0-458d-ac87-ba1107954798" (UID: "93947031-85b0-458d-ac87-ba1107954798"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:51:55.029159 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.029065    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-service-ca\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:55.029159 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.029104    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxmn\" (UniqueName: \"kubernetes.io/projected/93947031-85b0-458d-ac87-ba1107954798-kube-api-access-ctxmn\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:55.029159 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.029119    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-oauth-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:55.029159 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.029132    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93947031-85b0-458d-ac87-ba1107954798-oauth-serving-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:55.029159 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.029145    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93947031-85b0-458d-ac87-ba1107954798-console-serving-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:51:55.574771 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.574735    2569 generic.go:356] "Generic (PLEG): container finished" podID="93947031-85b0-458d-ac87-ba1107954798" containerID="3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0" exitCode=0
Jul 13 18:51:55.575268 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.574818    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-599d97cf86-6vrtz"
Jul 13 18:51:55.575268 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.574839    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599d97cf86-6vrtz" event={"ID":"93947031-85b0-458d-ac87-ba1107954798","Type":"ContainerDied","Data":"3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0"}
Jul 13 18:51:55.575268 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.574936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-599d97cf86-6vrtz" event={"ID":"93947031-85b0-458d-ac87-ba1107954798","Type":"ContainerDied","Data":"a696c5c5e616662f761ad1e6bc6e47e7ac66977749755395c136e5770d2b6b1a"}
Jul 13 18:51:55.575268 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.574961    2569 scope.go:117] "RemoveContainer" containerID="3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0"
Jul 13 18:51:55.583614 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.583596    2569 scope.go:117] "RemoveContainer" containerID="3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0"
Jul 13 18:51:55.583904 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:51:55.583870    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0\": container with ID starting with 3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0 not found: ID does not exist" containerID="3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0"
Jul 13 18:51:55.584011 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.583915    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0"} err="failed to get container status \"3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0\": rpc error: code = NotFound desc = could not find container \"3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0\": container with ID starting with 3962da3e7d852f4560260445965528f8fb13e895047637cd337031cce590fca0 not found: ID does not exist"
Jul 13 18:51:55.601255 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.601221    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-599d97cf86-6vrtz"]
Jul 13 18:51:55.603507 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:55.603480    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-599d97cf86-6vrtz"]
Jul 13 18:51:56.046275 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:51:56.046240    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93947031-85b0-458d-ac87-ba1107954798" path="/var/lib/kubelet/pods/93947031-85b0-458d-ac87-ba1107954798/volumes"
Jul 13 18:52:15.546247 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:15.546209    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:15.566029 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:15.566002    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:15.667605 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:15.667579    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:18.336589 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:18.336550    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:18.395662 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:18.395632    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:25.148227 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:25.148193    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:25.221915 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:25.221885    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:29.457596 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.457556    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:52:29.458048 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.457834    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="alertmanager" containerID="cri-o://d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1" gracePeriod=120
Jul 13 18:52:29.458048 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.457913    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy-metric" containerID="cri-o://ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed" gracePeriod=120
Jul 13 18:52:29.458048 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.457913    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy-web" containerID="cri-o://10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739" gracePeriod=120
Jul 13 18:52:29.458048 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.457960    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy" containerID="cri-o://adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1" gracePeriod=120
Jul 13 18:52:29.458048 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.458021    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="config-reloader" containerID="cri-o://eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb" gracePeriod=120
Jul 13 18:52:29.458393 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.457971    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="prom-label-proxy" containerID="cri-o://d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd" gracePeriod=120
Jul 13 18:52:29.705080 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.705046    2569 generic.go:356] "Generic (PLEG): container finished" podID="692933a6-f767-4df4-b6bb-7662eb17a867" containerID="d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd" exitCode=0
Jul 13 18:52:29.705080 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.705071    2569 generic.go:356] "Generic (PLEG): container finished" podID="692933a6-f767-4df4-b6bb-7662eb17a867" containerID="adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1" exitCode=0
Jul 13 18:52:29.705080 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.705079    2569 generic.go:356] "Generic (PLEG): container finished" podID="692933a6-f767-4df4-b6bb-7662eb17a867" containerID="eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb" exitCode=0
Jul 13 18:52:29.705080 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.705086    2569 generic.go:356] "Generic (PLEG): container finished" podID="692933a6-f767-4df4-b6bb-7662eb17a867" containerID="d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1" exitCode=0
Jul 13 18:52:29.705319 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.705121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerDied","Data":"d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd"}
Jul 13 18:52:29.705319 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.705213    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerDied","Data":"adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1"}
Jul 13 18:52:29.705319 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.705241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerDied","Data":"eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb"}
Jul 13 18:52:29.705319 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:29.705266    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerDied","Data":"d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1"}
Jul 13 18:52:30.707473 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.707446    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:52:30.711028 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.711003    2569 generic.go:356] "Generic (PLEG): container finished" podID="692933a6-f767-4df4-b6bb-7662eb17a867" containerID="ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed" exitCode=0
Jul 13 18:52:30.711028 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.711022    2569 generic.go:356] "Generic (PLEG): container finished" podID="692933a6-f767-4df4-b6bb-7662eb17a867" containerID="10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739" exitCode=0
Jul 13 18:52:30.711205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.711076    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:52:30.711205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.711081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerDied","Data":"ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed"}
Jul 13 18:52:30.711205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.711150    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerDied","Data":"10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739"}
Jul 13 18:52:30.711205 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.711177    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"692933a6-f767-4df4-b6bb-7662eb17a867","Type":"ContainerDied","Data":"25dcaa46269121151d4277a1cfe4d2c0158becca6dc80e7eb20f1eb22a9e80aa"}
Jul 13 18:52:30.711422 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.711208    2569 scope.go:117] "RemoveContainer" containerID="d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd"
Jul 13 18:52:30.718226 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.718212    2569 scope.go:117] "RemoveContainer" containerID="ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed"
Jul 13 18:52:30.724700 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.724685    2569 scope.go:117] "RemoveContainer" containerID="adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1"
Jul 13 18:52:30.728811 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.728792    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-main-db\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.728898 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.728824    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-web-config\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.728898 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.728849    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-cluster-tls-config\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729013 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.728902    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-metric\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729013 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.728933    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729013 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.728959    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np4bg\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-kube-api-access-np4bg\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729013 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.728992    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-main-tls\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729210 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.729026    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-config-volume\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729210 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.729062    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-web\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729210 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.729097    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-config-out\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729210 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.729107    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:52:30.729210 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.729125    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-tls-assets\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729210 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.729161    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-trusted-ca-bundle\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.729507 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.729402    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-main-db\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.729885 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.729775    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:30.732405 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.732369    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-config-out" (OuterVolumeSpecName: "config-out") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:52:30.732909 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.732860    2569 scope.go:117] "RemoveContainer" containerID="10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739"
Jul 13 18:52:30.733746 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.733715    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:30.733840 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.733814    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-config-volume" (OuterVolumeSpecName: "config-volume") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:30.733967 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.733911    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:30.734570 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.734544    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:52:30.735347 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.735299    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:30.735446 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.735380    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-kube-api-access-np4bg" (OuterVolumeSpecName: "kube-api-access-np4bg") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "kube-api-access-np4bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:52:30.735446 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.735394    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:30.737731 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.737619    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:30.743873 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.743855    2569 scope.go:117] "RemoveContainer" containerID="eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb"
Jul 13 18:52:30.744365 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.744338    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-web-config" (OuterVolumeSpecName: "web-config") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:30.751811 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.751791    2569 scope.go:117] "RemoveContainer" containerID="d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1"
Jul 13 18:52:30.758779 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.758759    2569 scope.go:117] "RemoveContainer" containerID="66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487"
Jul 13 18:52:30.766313 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.766293    2569 scope.go:117] "RemoveContainer" containerID="d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd"
Jul 13 18:52:30.766627 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:30.766602    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd\": container with ID starting with d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd not found: ID does not exist" containerID="d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd"
Jul 13 18:52:30.766730 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.766639    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd"} err="failed to get container status \"d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd\": rpc error: code = NotFound desc = could not find container \"d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd\": container with ID starting with d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd not found: ID does not exist"
Jul 13 18:52:30.766730 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.766663    2569 scope.go:117] "RemoveContainer" containerID="ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed"
Jul 13 18:52:30.767022 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:30.766955    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed\": container with ID starting with ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed not found: ID does not exist" containerID="ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed"
Jul 13 18:52:30.767022 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.766993    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed"} err="failed to get container status \"ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed\": rpc error: code = NotFound desc = could not find container \"ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed\": container with ID starting with ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed not found: ID does not exist"
Jul 13 18:52:30.767022 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.767018    2569 scope.go:117] "RemoveContainer" containerID="adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1"
Jul 13 18:52:30.767358 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:30.767307    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1\": container with ID starting with adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1 not found: ID does not exist" containerID="adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1"
Jul 13 18:52:30.767432 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.767349    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1"} err="failed to get container status \"adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1\": rpc error: code = NotFound desc = could not find container \"adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1\": container with ID starting with adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1 not found: ID does not exist"
Jul 13 18:52:30.767432 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.767372    2569 scope.go:117] "RemoveContainer" containerID="10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739"
Jul 13 18:52:30.767737 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:30.767708    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739\": container with ID starting with 10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739 not found: ID does not exist" containerID="10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739"
Jul 13 18:52:30.767840 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.767741    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739"} err="failed to get container status \"10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739\": rpc error: code = NotFound desc = could not find container \"10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739\": container with ID starting with 10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739 not found: ID does not exist"
Jul 13 18:52:30.767840 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.767765    2569 scope.go:117] "RemoveContainer" containerID="eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb"
Jul 13 18:52:30.768079 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:30.768056    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb\": container with ID starting with eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb not found: ID does not exist" containerID="eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb"
Jul 13 18:52:30.768142 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.768086    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb"} err="failed to get container status \"eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb\": rpc error: code = NotFound desc = could not find container \"eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb\": container with ID starting with eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb not found: ID does not exist"
Jul 13 18:52:30.768142 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.768105    2569 scope.go:117] "RemoveContainer" containerID="d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1"
Jul 13 18:52:30.768438 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:30.768414    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1\": container with ID starting with d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1 not found: ID does not exist" containerID="d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1"
Jul 13 18:52:30.768598 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.768444    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1"} err="failed to get container status \"d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1\": rpc error: code = NotFound desc = could not find container \"d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1\": container with ID starting with d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1 not found: ID does not exist"
Jul 13 18:52:30.768598 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.768467    2569 scope.go:117] "RemoveContainer" containerID="66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487"
Jul 13 18:52:30.768802 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:30.768776    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487\": container with ID starting with 66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487 not found: ID does not exist" containerID="66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487"
Jul 13 18:52:30.768878 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.768809    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487"} err="failed to get container status \"66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487\": rpc error: code = NotFound desc = could not find container \"66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487\": container with ID starting with 66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487 not found: ID does not exist"
Jul 13 18:52:30.768878 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.768840    2569 scope.go:117] "RemoveContainer" containerID="d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd"
Jul 13 18:52:30.769172 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.769124    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd"} err="failed to get container status \"d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd\": rpc error: code = NotFound desc = could not find container \"d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd\": container with ID starting with d1bc95d5f1ea762c39c008cdb641b7ba1fa218333a03d2a041ac186d8b6b8dcd not found: ID does not exist"
Jul 13 18:52:30.769261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.769174    2569 scope.go:117] "RemoveContainer" containerID="ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.770743    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed"} err="failed to get container status \"ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed\": rpc error: code = NotFound desc = could not find container \"ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed\": container with ID starting with ba6cc7403fc6f90edf8028717eb6ec8efd6866c55514dd1610b4c56abc5411ed not found: ID does not exist"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.770771    2569 scope.go:117] "RemoveContainer" containerID="adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.771007    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1"} err="failed to get container status \"adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1\": rpc error: code = NotFound desc = could not find container \"adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1\": container with ID starting with adca229c24f84e93048db4ee13942a8afa8f7ece2c7198009b3c4a90f041fac1 not found: ID does not exist"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.771028    2569 scope.go:117] "RemoveContainer" containerID="10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.771247    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739"} err="failed to get container status \"10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739\": rpc error: code = NotFound desc = could not find container \"10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739\": container with ID starting with 10c1ade4d70dbafd44433b84478a24fcc88d4cbfe3a33f013ac0fbde0d5e8739 not found: ID does not exist"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.771267    2569 scope.go:117] "RemoveContainer" containerID="eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.771476    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb"} err="failed to get container status \"eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb\": rpc error: code = NotFound desc = could not find container \"eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb\": container with ID starting with eae3519af249a7ef9d00a9751216e3eca0ccac700abf12873a7aaefbe04974eb not found: ID does not exist"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.771499    2569 scope.go:117] "RemoveContainer" containerID="d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.771727    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1"} err="failed to get container status \"d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1\": rpc error: code = NotFound desc = could not find container \"d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1\": container with ID starting with d4c7b56632f2bdd212a21743ad60a4b129082a171c1b176ba94ef89e042a23a1 not found: ID does not exist"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.771748    2569 scope.go:117] "RemoveContainer" containerID="66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487"
Jul 13 18:52:30.771940 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.771932    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487"} err="failed to get container status \"66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487\": rpc error: code = NotFound desc = could not find container \"66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487\": container with ID starting with 66b47894804f4442bc6ae494e4b91e257cc67e3e13a21126297be7ad32134487 not found: ID does not exist"
Jul 13 18:52:30.829743 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829663    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-metrics-client-ca\") pod \"692933a6-f767-4df4-b6bb-7662eb17a867\" (UID: \"692933a6-f767-4df4-b6bb-7662eb17a867\") "
Jul 13 18:52:30.829871 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829807    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-web-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.829871 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829819    2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-cluster-tls-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.829871 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829829    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.829871 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829839    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.829871 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829848    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np4bg\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-kube-api-access-np4bg\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.829871 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829857    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-main-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.829871 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829865    2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-config-volume\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.829871 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829874    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/692933a6-f767-4df4-b6bb-7662eb17a867-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.830164 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829883    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/692933a6-f767-4df4-b6bb-7662eb17a867-config-out\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.830164 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829890    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/692933a6-f767-4df4-b6bb-7662eb17a867-tls-assets\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.830164 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829900    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:30.830164 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.829974    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "692933a6-f767-4df4-b6bb-7662eb17a867" (UID: "692933a6-f767-4df4-b6bb-7662eb17a867"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:30.930934 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:30.930903    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/692933a6-f767-4df4-b6bb-7662eb17a867-metrics-client-ca\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:31.036069 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:31.036038    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:52:31.041616 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:31.041592    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:52:32.044874 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:32.044833    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" path="/var/lib/kubelet/pods/692933a6-f767-4df4-b6bb-7662eb17a867/volumes"
Jul 13 18:52:33.136207 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:33.136178    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:33.218079 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:33.218047    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:33.799255 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:33.799221    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:52:33.799609 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:33.799576    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="prometheus" containerID="cri-o://82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce" gracePeriod=600
Jul 13 18:52:33.800143 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:33.799778    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://29274eab4d8dd470b41ee1c2115afa388aa498a890ff2489db00419c886a4eff" gracePeriod=600
Jul 13 18:52:33.800416 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:33.799787    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="thanos-sidecar" containerID="cri-o://7e09683d89a4c2a8ca4c9f98933e4a0447f048cd1dd603cbf01157dd29c641c5" gracePeriod=600
Jul 13 18:52:33.801075 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:33.799858    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy" containerID="cri-o://465b67ac295fa53c7d82a20591b546e6e10e6fd710c27afd620a56276917bdba" gracePeriod=600
Jul 13 18:52:33.801367 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:33.799876    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="config-reloader" containerID="cri-o://f8aa9f85452d26368f6636c21b9cab2511c95fac77b243c65abbb30d90227fce" gracePeriod=600
Jul 13 18:52:33.801458 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:33.800030    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy-web" containerID="cri-o://03d29e0d18cabac1b0eb5e765781bdd1be3480d742f6397525808d76c3ccdd6a" gracePeriod=600
Jul 13 18:52:34.732078 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:34.732040    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerID="03d29e0d18cabac1b0eb5e765781bdd1be3480d742f6397525808d76c3ccdd6a" exitCode=0
Jul 13 18:52:34.732078 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:34.732071    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerID="7e09683d89a4c2a8ca4c9f98933e4a0447f048cd1dd603cbf01157dd29c641c5" exitCode=0
Jul 13 18:52:34.732078 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:34.732080    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerID="f8aa9f85452d26368f6636c21b9cab2511c95fac77b243c65abbb30d90227fce" exitCode=0
Jul 13 18:52:34.732548 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:34.732118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerDied","Data":"03d29e0d18cabac1b0eb5e765781bdd1be3480d742f6397525808d76c3ccdd6a"}
Jul 13 18:52:34.732548 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:34.732200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerDied","Data":"7e09683d89a4c2a8ca4c9f98933e4a0447f048cd1dd603cbf01157dd29c641c5"}
Jul 13 18:52:34.732548 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:34.732243    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerDied","Data":"f8aa9f85452d26368f6636c21b9cab2511c95fac77b243c65abbb30d90227fce"}
Jul 13 18:52:35.654032 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:35.653991    2569 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 13 18:52:35.655037 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:35.655006    2569 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 13 18:52:35.655872 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:35.655841    2569 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 13 18:52:35.655974 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:35.655882    2569 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="prometheus" probeResult="unknown"
Jul 13 18:52:35.740473 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:35.740438    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerID="29274eab4d8dd470b41ee1c2115afa388aa498a890ff2489db00419c886a4eff" exitCode=0
Jul 13 18:52:35.740473 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:35.740462    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerID="465b67ac295fa53c7d82a20591b546e6e10e6fd710c27afd620a56276917bdba" exitCode=0
Jul 13 18:52:35.740983 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:35.740511    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerDied","Data":"29274eab4d8dd470b41ee1c2115afa388aa498a890ff2489db00419c886a4eff"}
Jul 13 18:52:35.740983 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:35.740593    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerDied","Data":"465b67ac295fa53c7d82a20591b546e6e10e6fd710c27afd620a56276917bdba"}
Jul 13 18:52:37.083366 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:37.083304    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:37.183145 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:37.183106    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:40.361182 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:40.361145    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:40.461115 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:40.461086    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:40.653430 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:40.653389    2569 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 13 18:52:40.654376 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:40.654345    2569 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 13 18:52:40.655209 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:40.655178    2569 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 13 18:52:40.655293 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:52:40.655226    2569 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="prometheus" probeResult="unknown"
Jul 13 18:52:41.766457 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:41.766426    2569 generic.go:356] "Generic (PLEG): container finished" podID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerID="82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce" exitCode=0
Jul 13 18:52:41.766804 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:41.766463    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerDied","Data":"82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce"}
Jul 13 18:52:42.351545 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.351508    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.421626 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421592    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-db\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.421797 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421637    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config-out\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.421797 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421656    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-1\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.421797 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421683    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-metrics-client-ca\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.421797 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.421797 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421739    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-kube-rbac-proxy\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.421797 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421766    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-thanos-prometheus-http-client-file\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421799    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-0\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421829    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-tls-assets\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421866    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-kubelet-serving-ca-bundle\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421896    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-2\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421922    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-tls\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.421966    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-serving-certs-ca-bundle\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422000    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-metrics-client-certs\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422023    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422040    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422069    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422110 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422105    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-web-config\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422681 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422127    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-grpc-tls\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422681 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422163    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwps4\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-kube-api-access-cwps4\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422681 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422188    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-trusted-ca-bundle\") pod \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\" (UID: \"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0\") "
Jul 13 18:52:42.422681 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422452    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.422681 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422570    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:42.422681 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422603    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:42.422985 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422806    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:52:42.423040 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.422981    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:42.423219 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.423192    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:42.423586 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.423292    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:42.425333 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.425305    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:52:42.427498 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.426028    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config-out" (OuterVolumeSpecName: "config-out") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:52:42.427498 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.426272    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:42.427498 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.426511    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:52:42.427738 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.427639    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:42.429710 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.429668    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:42.429995 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.429917    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:42.434105 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.434063    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:42.434512 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.434474    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-kube-api-access-cwps4" (OuterVolumeSpecName: "kube-api-access-cwps4") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "kube-api-access-cwps4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:52:42.434629 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.434599    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config" (OuterVolumeSpecName: "config") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:42.435580 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.435556    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:42.436351 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.436325    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:42.441833 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.441811    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-web-config" (OuterVolumeSpecName: "web-config") pod "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" (UID: "dcf545e0-3407-41c2-beb0-d0f6e1e89eb0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523377    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523412    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523428    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523441    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-metrics-client-certs\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523454    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523469    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523509    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-web-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523523    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-grpc-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523558    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwps4\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-kube-api-access-cwps4\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523571    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523583    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-db\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523595    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-config-out\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523607    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-metrics-client-ca\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523621    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523633    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-secret-kube-rbac-proxy\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523646    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523658    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523671    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-tls-assets\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.525452 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.523683    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:52:42.773185 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.773153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcf545e0-3407-41c2-beb0-d0f6e1e89eb0","Type":"ContainerDied","Data":"727c3f7f4c97e13cc142e5c5daeb9107c7f3abb44a408fe95f554888ef591c21"}
Jul 13 18:52:42.773559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.773164    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.773559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.773263    2569 scope.go:117] "RemoveContainer" containerID="29274eab4d8dd470b41ee1c2115afa388aa498a890ff2489db00419c886a4eff"
Jul 13 18:52:42.781448 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.781431    2569 scope.go:117] "RemoveContainer" containerID="465b67ac295fa53c7d82a20591b546e6e10e6fd710c27afd620a56276917bdba"
Jul 13 18:52:42.788693 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.788679    2569 scope.go:117] "RemoveContainer" containerID="03d29e0d18cabac1b0eb5e765781bdd1be3480d742f6397525808d76c3ccdd6a"
Jul 13 18:52:42.795177 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.795162    2569 scope.go:117] "RemoveContainer" containerID="7e09683d89a4c2a8ca4c9f98933e4a0447f048cd1dd603cbf01157dd29c641c5"
Jul 13 18:52:42.799881 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.799862    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:52:42.802199 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.802179    2569 scope.go:117] "RemoveContainer" containerID="f8aa9f85452d26368f6636c21b9cab2511c95fac77b243c65abbb30d90227fce"
Jul 13 18:52:42.806375 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.806356    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:52:42.808952 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.808935    2569 scope.go:117] "RemoveContainer" containerID="82af7004dafee62ec47b63e23d0c9ded14878dabf057cf966e8330f10e0c64ce"
Jul 13 18:52:42.816113 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.816098    2569 scope.go:117] "RemoveContainer" containerID="1356fdcdb21fc801d12e14512db5f42bc64a7a320412231c227deae845e06590"
Jul 13 18:52:42.838102 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838078    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:52:42.838331 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838319    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy-metric"
Jul 13 18:52:42.838371 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838332    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy-metric"
Jul 13 18:52:42.838371 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838343    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="thanos-sidecar"
Jul 13 18:52:42.838371 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838349    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="thanos-sidecar"
Jul 13 18:52:42.838371 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838358    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93947031-85b0-458d-ac87-ba1107954798" containerName="console"
Jul 13 18:52:42.838371 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838363    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="93947031-85b0-458d-ac87-ba1107954798" containerName="console"
Jul 13 18:52:42.838371 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838369    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838374    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838381    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="prometheus"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838386    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="prometheus"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838393    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy-web"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838398    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy-web"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838404    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="init-config-reloader"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838409    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="init-config-reloader"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838415    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy-web"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838420    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy-web"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838427    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="config-reloader"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838431    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="config-reloader"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838439    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="init-config-reloader"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838444    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="init-config-reloader"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838451    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838455    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838461    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy-thanos"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838466    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy-thanos"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838472    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="config-reloader"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838476    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="config-reloader"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838485    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" containerName="console"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838490    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" containerName="console"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838496    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="alertmanager"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838500    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="alertmanager"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838506    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="prom-label-proxy"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838511    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="prom-label-proxy"
Jul 13 18:52:42.838559 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838565    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy-web"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838574    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy-metric"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838580    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy-thanos"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838585    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="prom-label-proxy"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838591    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy-web"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838596    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="thanos-sidecar"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838601    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="alertmanager"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838606    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="prometheus"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838611    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="93947031-85b0-458d-ac87-ba1107954798" containerName="console"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838615    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="kube-rbac-proxy"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838621    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="config-reloader"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838626    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac0d6c1c-76f5-4da4-bbe3-2de78f4735f4" containerName="console"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838630    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="692933a6-f767-4df4-b6bb-7662eb17a867" containerName="kube-rbac-proxy"
Jul 13 18:52:42.839261 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.838635    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" containerName="config-reloader"
Jul 13 18:52:42.841900 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.841885    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.844793 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.844754    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 13 18:52:42.844982 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.844961    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 13 18:52:42.845077 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.845063    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hqf4n\""
Jul 13 18:52:42.845362 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.845316    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 13 18:52:42.845727 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.845681    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 13 18:52:42.845842 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.845812    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 13 18:52:42.845939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.845838    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 13 18:52:42.845939 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.845897    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 13 18:52:42.846049 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.846002    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 13 18:52:42.846224 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.846189    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 13 18:52:42.846315 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.846222    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 13 18:52:42.846507 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.846488    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eef1nuotvtqv\""
Jul 13 18:52:42.846633 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.846585    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 13 18:52:42.846690 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.846645    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 13 18:52:42.849508 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.849339    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 13 18:52:42.850927 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.850901    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 13 18:52:42.853484 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.853466    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 13 18:52:42.868826 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.868799    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:52:42.926808 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.926779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.926808 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.926812    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927023 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.926833    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927023 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.926852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927023 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.926913    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927023 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.926985    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b11fcb47-48c1-4188-809b-035b5deb68d7-config-out\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927023 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927009    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927168 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927168 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927078    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927168 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927168 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927139    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927168 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927156    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927322 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b11fcb47-48c1-4188-809b-035b5deb68d7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927322 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927195    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927322 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927221    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-config\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927322 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927235    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-web-config\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927322 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927256    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927322 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927278    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptldf\" (UniqueName: \"kubernetes.io/projected/b11fcb47-48c1-4188-809b-035b5deb68d7-kube-api-access-ptldf\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927322 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927314    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:42.927522 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:42.927352    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028370 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028623 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028623 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028405    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028623 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028423    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028623 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028475    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b11fcb47-48c1-4188-809b-035b5deb68d7-config-out\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028623 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028510    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028883 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028678    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028883 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028714    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028883 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028754    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028883 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028883 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028883 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028835    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b11fcb47-48c1-4188-809b-035b5deb68d7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.028883 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028864    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-config\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-web-config\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.028975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptldf\" (UniqueName: \"kubernetes.io/projected/b11fcb47-48c1-4188-809b-035b5deb68d7-kube-api-access-ptldf\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.029028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.029072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.029163    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.029207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029323 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.029238    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.029767 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.029469    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.030328 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.030303    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.031691 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.031632    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b11fcb47-48c1-4188-809b-035b5deb68d7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.031802 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.031705    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.031888 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.031867    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.032302 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.032274    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.032662 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.032481    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-web-config\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.032988 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.032774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.032988 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.032784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.032988 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.032928    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b11fcb47-48c1-4188-809b-035b5deb68d7-config-out\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.033478 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.033451    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b11fcb47-48c1-4188-809b-035b5deb68d7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.034132 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.033915    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.034132 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.034072    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.034277 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.034222    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.034376 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.034357    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-config\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.034953 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.034934    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b11fcb47-48c1-4188-809b-035b5deb68d7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.038364 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.038343    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptldf\" (UniqueName: \"kubernetes.io/projected/b11fcb47-48c1-4188-809b-035b5deb68d7-kube-api-access-ptldf\") pod \"prometheus-k8s-0\" (UID: \"b11fcb47-48c1-4188-809b-035b5deb68d7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.153277 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.153230    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:52:43.289154 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.289082    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 13 18:52:43.291434 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:52:43.291411    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11fcb47_48c1_4188_809b_035b5deb68d7.slice/crio-0c39dc69aed84f4fe12a3c06079f435813a8fd20ddf0a34c95968375893d8f85 WatchSource:0}: Error finding container 0c39dc69aed84f4fe12a3c06079f435813a8fd20ddf0a34c95968375893d8f85: Status 404 returned error can't find the container with id 0c39dc69aed84f4fe12a3c06079f435813a8fd20ddf0a34c95968375893d8f85
Jul 13 18:52:43.542311 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.542222    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c4fd9f694-zb6lp"]
Jul 13 18:52:43.652189 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.652154    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:43.753426 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.753392    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:43.779562 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.779511    2569 generic.go:356] "Generic (PLEG): container finished" podID="b11fcb47-48c1-4188-809b-035b5deb68d7" containerID="b65fe4c8025b357ec0ea642e9e84af4f3d40ab82f451b0df443fe56c62282057" exitCode=0
Jul 13 18:52:43.779880 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.779571    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b11fcb47-48c1-4188-809b-035b5deb68d7","Type":"ContainerDied","Data":"b65fe4c8025b357ec0ea642e9e84af4f3d40ab82f451b0df443fe56c62282057"}
Jul 13 18:52:43.779880 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.779622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b11fcb47-48c1-4188-809b-035b5deb68d7","Type":"ContainerStarted","Data":"0c39dc69aed84f4fe12a3c06079f435813a8fd20ddf0a34c95968375893d8f85"}
Jul 13 18:52:43.834487 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:43.834450    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:52:44.045680 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:44.045139    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf545e0-3407-41c2-beb0-d0f6e1e89eb0" path="/var/lib/kubelet/pods/dcf545e0-3407-41c2-beb0-d0f6e1e89eb0/volumes"
Jul 13 18:52:44.787765 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:44.787726    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b11fcb47-48c1-4188-809b-035b5deb68d7","Type":"ContainerStarted","Data":"4bfd00bc9c0fe6d20dea38450c0977d1165b7a73f5e379a51c563003a46d3235"}
Jul 13 18:52:44.788247 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:44.787804    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b11fcb47-48c1-4188-809b-035b5deb68d7","Type":"ContainerStarted","Data":"86ee7e56fae71284d2a9960b0ae7d75dff7f020d72c4b61089d5b706ec270b82"}
Jul 13 18:52:44.788247 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:44.787846    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b11fcb47-48c1-4188-809b-035b5deb68d7","Type":"ContainerStarted","Data":"b88af9ac82ecc67f4d2a5f9f78b5d3f1e16a33ba5866004264afa094a59377b6"}
Jul 13 18:52:44.788247 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:44.787885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b11fcb47-48c1-4188-809b-035b5deb68d7","Type":"ContainerStarted","Data":"34c937b6c9951f2863cdb763df38ecd5d1bde7154f96c0dd29c6cf16b1365185"}
Jul 13 18:52:44.788247 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:44.787927    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b11fcb47-48c1-4188-809b-035b5deb68d7","Type":"ContainerStarted","Data":"c33a78b95fc2ee7afa437b5d188b60f5c9ead3ce590242d40daf356f5c47628a"}
Jul 13 18:52:44.788247 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:44.787974    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b11fcb47-48c1-4188-809b-035b5deb68d7","Type":"ContainerStarted","Data":"e42af9f6548e9f8ac2b1b4d78cee3221f07df20b8f78ee3bcf6f79ee4d528e04"}
Jul 13 18:52:48.153345 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:52:48.153309    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:08.561980 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.561931    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c4fd9f694-zb6lp" podUID="d12a3724-c4f4-4146-8c0e-2ef9d9130220" containerName="console" containerID="cri-o://9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440" gracePeriod=15
Jul 13 18:53:08.796898 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.796875    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:53:08.819214 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.819114    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=26.798027138 podStartE2EDuration="26.819093054s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.065916ms" isStatefulPod=false podCreationTimestamp="2026-07-13 18:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:52:44.842712475 +0000 UTC m=+156.948316096" watchObservedRunningTime="2026-07-13 18:53:08.819093054 +0000 UTC m=+180.924696676"
Jul 13 18:53:08.882081 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.882046    2569 generic.go:356] "Generic (PLEG): container finished" podID="d12a3724-c4f4-4146-8c0e-2ef9d9130220" containerID="9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440" exitCode=0
Jul 13 18:53:08.882224 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.882096    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4fd9f694-zb6lp"
Jul 13 18:53:08.882224 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.882138    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4fd9f694-zb6lp" event={"ID":"d12a3724-c4f4-4146-8c0e-2ef9d9130220","Type":"ContainerDied","Data":"9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440"}
Jul 13 18:53:08.882224 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.882212    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4fd9f694-zb6lp" event={"ID":"d12a3724-c4f4-4146-8c0e-2ef9d9130220","Type":"ContainerDied","Data":"939102c8ef57a0bd078c9e5b003ee98b6515e1ace7341835c089a577f3d75661"}
Jul 13 18:53:08.882395 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.882241    2569 scope.go:117] "RemoveContainer" containerID="9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440"
Jul 13 18:53:08.889973 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.889956    2569 scope.go:117] "RemoveContainer" containerID="9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440"
Jul 13 18:53:08.890262 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:53:08.890243    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440\": container with ID starting with 9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440 not found: ID does not exist" containerID="9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440"
Jul 13 18:53:08.890322 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.890272    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440"} err="failed to get container status \"9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440\": rpc error: code = NotFound desc = could not find container \"9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440\": container with ID starting with 9d293546a10e7b1d4024da7a35e7dff918237f39096563bda9e5c2ec3ff3c440 not found: ID does not exist"
Jul 13 18:53:08.949603 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.949579    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-trusted-ca-bundle\") pod \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") "
Jul 13 18:53:08.949709 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.949607    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-oauth-config\") pod \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") "
Jul 13 18:53:08.949709 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.949641    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-service-ca\") pod \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") "
Jul 13 18:53:08.949709 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.949666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5cmf\" (UniqueName: \"kubernetes.io/projected/d12a3724-c4f4-4146-8c0e-2ef9d9130220-kube-api-access-z5cmf\") pod \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") "
Jul 13 18:53:08.949709 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.949709    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-oauth-serving-cert\") pod \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") "
Jul 13 18:53:08.949917 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.949771    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-config\") pod \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") "
Jul 13 18:53:08.949917 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.949787    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-serving-cert\") pod \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\" (UID: \"d12a3724-c4f4-4146-8c0e-2ef9d9130220\") "
Jul 13 18:53:08.950111 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.950076    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d12a3724-c4f4-4146-8c0e-2ef9d9130220" (UID: "d12a3724-c4f4-4146-8c0e-2ef9d9130220"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:08.950185 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.950114    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-service-ca" (OuterVolumeSpecName: "service-ca") pod "d12a3724-c4f4-4146-8c0e-2ef9d9130220" (UID: "d12a3724-c4f4-4146-8c0e-2ef9d9130220"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:08.950255 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.950231    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d12a3724-c4f4-4146-8c0e-2ef9d9130220" (UID: "d12a3724-c4f4-4146-8c0e-2ef9d9130220"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:08.950347 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.950313    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-config" (OuterVolumeSpecName: "console-config") pod "d12a3724-c4f4-4146-8c0e-2ef9d9130220" (UID: "d12a3724-c4f4-4146-8c0e-2ef9d9130220"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:08.951918 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.951889    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d12a3724-c4f4-4146-8c0e-2ef9d9130220" (UID: "d12a3724-c4f4-4146-8c0e-2ef9d9130220"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:08.951997 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.951932    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d12a3724-c4f4-4146-8c0e-2ef9d9130220" (UID: "d12a3724-c4f4-4146-8c0e-2ef9d9130220"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:08.951997 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:08.951932    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12a3724-c4f4-4146-8c0e-2ef9d9130220-kube-api-access-z5cmf" (OuterVolumeSpecName: "kube-api-access-z5cmf") pod "d12a3724-c4f4-4146-8c0e-2ef9d9130220" (UID: "d12a3724-c4f4-4146-8c0e-2ef9d9130220"). InnerVolumeSpecName "kube-api-access-z5cmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:53:09.050660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:09.050630    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-oauth-serving-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:53:09.050660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:09.050654    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:53:09.050660 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:09.050663    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-serving-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:53:09.050861 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:09.050671    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-trusted-ca-bundle\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:53:09.050861 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:09.050679    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d12a3724-c4f4-4146-8c0e-2ef9d9130220-console-oauth-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:53:09.050861 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:09.050687    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3724-c4f4-4146-8c0e-2ef9d9130220-service-ca\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:53:09.050861 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:09.050704    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5cmf\" (UniqueName: \"kubernetes.io/projected/d12a3724-c4f4-4146-8c0e-2ef9d9130220-kube-api-access-z5cmf\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:53:09.203708 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:09.203682    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c4fd9f694-zb6lp"]
Jul 13 18:53:09.207175 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:09.207147    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c4fd9f694-zb6lp"]
Jul 13 18:53:10.052329 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:10.052295    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12a3724-c4f4-4146-8c0e-2ef9d9130220" path="/var/lib/kubelet/pods/d12a3724-c4f4-4146-8c0e-2ef9d9130220/volumes"
Jul 13 18:53:14.380495 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:14.380461    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:16.426134 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:16.426102    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:53:43.154043 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:43.154005    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:43.169083 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:43.169057    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:53:44.020456 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:53:44.020427    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 13 18:54:05.312086 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:54:05.312052    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:40.779301 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:54:40.779221    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:40.834320 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:54:40.834275    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:41.216389 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:54:41.216356    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:41.306093 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:54:41.306060    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:54:41.411013 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:54:41.410981    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:08.744675 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:08.744633    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:09.871366 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:09.871340    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 18:55:09.871796 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:09.871378    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 18:55:09.874182 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:09.874164    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 18:55:10.035757 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:10.035607    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:13.666385 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:13.666349    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:13.947736 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:13.947660    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:14.168592 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:14.168560    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:14.378943 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:14.378836    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:14.549418 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:14.549388    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:14.779293 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:14.779264    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:14.885387 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:14.885353    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:14.984056 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:14.983981    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:15.097398 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:15.097293    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:15.185292 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:15.185260    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:15.289811 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:15.289780    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:15.385616 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:15.385585    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:55:49.230413 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:55:49.230379    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:23.402374 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:23.402340    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:23.467807 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:23.467760    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:23.643308 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:23.643274    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:24.826322 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:24.826269    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:24.926664 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:24.926622    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:30.467272 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:30.467240    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:30.571860 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:30.571810    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:30.668393 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:30.668356    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:31.218892 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:31.218862    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:31.318039 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:31.318006    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:41.617500 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:41.617466    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:41.719762 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:41.719730    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:41.795316 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:41.795281    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:56:58.823806 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:56:58.823770    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:35.661449 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.661372    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:35.708785 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.708751    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-swzc7"]
Jul 13 18:57:35.709093 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.709075    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12a3724-c4f4-4146-8c0e-2ef9d9130220" containerName="console"
Jul 13 18:57:35.709169 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.709096    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12a3724-c4f4-4146-8c0e-2ef9d9130220" containerName="console"
Jul 13 18:57:35.709169 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.709164    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12a3724-c4f4-4146-8c0e-2ef9d9130220" containerName="console"
Jul 13 18:57:35.712116 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.712097    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:35.715351 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.715331    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 13 18:57:35.715520 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.715503    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 13 18:57:35.716088 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.716069    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 13 18:57:35.716162 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.716096    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-xqb6q\""
Jul 13 18:57:35.724664 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.724641    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-swzc7"]
Jul 13 18:57:35.755749 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.755722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56s5\" (UniqueName: \"kubernetes.io/projected/c2e0b847-946c-4ec7-acfd-a2945b244533-kube-api-access-r56s5\") pod \"model-serving-api-fd65d7d6-swzc7\" (UID: \"c2e0b847-946c-4ec7-acfd-a2945b244533\") " pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:35.755873 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.755772    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e0b847-946c-4ec7-acfd-a2945b244533-tls-certs\") pod \"model-serving-api-fd65d7d6-swzc7\" (UID: \"c2e0b847-946c-4ec7-acfd-a2945b244533\") " pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:35.762628 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.762588    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:35.856380 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.856339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56s5\" (UniqueName: \"kubernetes.io/projected/c2e0b847-946c-4ec7-acfd-a2945b244533-kube-api-access-r56s5\") pod \"model-serving-api-fd65d7d6-swzc7\" (UID: \"c2e0b847-946c-4ec7-acfd-a2945b244533\") " pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:35.856613 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.856400    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e0b847-946c-4ec7-acfd-a2945b244533-tls-certs\") pod \"model-serving-api-fd65d7d6-swzc7\" (UID: \"c2e0b847-946c-4ec7-acfd-a2945b244533\") " pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:35.856613 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:57:35.856501    2569 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 13 18:57:35.856613 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:57:35.856590    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e0b847-946c-4ec7-acfd-a2945b244533-tls-certs podName:c2e0b847-946c-4ec7-acfd-a2945b244533 nodeName:}" failed. No retries permitted until 2026-07-13 18:57:36.356569812 +0000 UTC m=+448.462173409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c2e0b847-946c-4ec7-acfd-a2945b244533-tls-certs") pod "model-serving-api-fd65d7d6-swzc7" (UID: "c2e0b847-946c-4ec7-acfd-a2945b244533") : secret "model-serving-api-tls" not found
Jul 13 18:57:35.865587 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.865563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56s5\" (UniqueName: \"kubernetes.io/projected/c2e0b847-946c-4ec7-acfd-a2945b244533-kube-api-access-r56s5\") pod \"model-serving-api-fd65d7d6-swzc7\" (UID: \"c2e0b847-946c-4ec7-acfd-a2945b244533\") " pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:35.992134 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:35.992049    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:36.114215 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:36.114181    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:36.359924 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:36.359831    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e0b847-946c-4ec7-acfd-a2945b244533-tls-certs\") pod \"model-serving-api-fd65d7d6-swzc7\" (UID: \"c2e0b847-946c-4ec7-acfd-a2945b244533\") " pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:36.362258 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:36.362227    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e0b847-946c-4ec7-acfd-a2945b244533-tls-certs\") pod \"model-serving-api-fd65d7d6-swzc7\" (UID: \"c2e0b847-946c-4ec7-acfd-a2945b244533\") " pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:36.551414 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:36.551378    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:36.622395 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:36.622310    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:36.662647 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:36.662618    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:57:36.760073 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:36.760037    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-swzc7"]
Jul 13 18:57:36.762988 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:57:36.762957    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e0b847_946c_4ec7_acfd_a2945b244533.slice/crio-572d7e7be2c987fd37dc49753c4800c2b7028722b753e5d8ac7685eb954054be WatchSource:0}: Error finding container 572d7e7be2c987fd37dc49753c4800c2b7028722b753e5d8ac7685eb954054be: Status 404 returned error can't find the container with id 572d7e7be2c987fd37dc49753c4800c2b7028722b753e5d8ac7685eb954054be
Jul 13 18:57:36.764866 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:36.764847    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:57:36.803641 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:36.803611    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-swzc7" event={"ID":"c2e0b847-946c-4ec7-acfd-a2945b244533","Type":"ContainerStarted","Data":"572d7e7be2c987fd37dc49753c4800c2b7028722b753e5d8ac7685eb954054be"}
Jul 13 18:57:39.818004 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:39.817972    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-swzc7" event={"ID":"c2e0b847-946c-4ec7-acfd-a2945b244533","Type":"ContainerStarted","Data":"33de5a1877d17046c4e917cfff3b96bef34780fe80fd1e6ad2f8757b2fbe7148"}
Jul 13 18:57:39.818372 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:39.818039    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:39.837276 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:39.837223    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-swzc7" podStartSLOduration=2.434376242 podStartE2EDuration="4.837202899s" totalImagesPullingTime="2.402826657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:57:39.83660782 +0000 UTC m=+451.942211440" watchObservedRunningTime="2026-07-13 18:57:39.837202899 +0000 UTC m=+451.942806519"
Jul 13 18:57:50.825538 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:50.825495    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-swzc7"
Jul 13 18:57:53.599665 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.599620    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-7sntz"]
Jul 13 18:57:53.602965 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.602948    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7sntz"
Jul 13 18:57:53.605717 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.605696    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-49p2v\""
Jul 13 18:57:53.605832 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.605767    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 13 18:57:53.609437 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.609402    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7sntz"]
Jul 13 18:57:53.700632 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.700604    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d6a6123-d931-4568-9ab9-14e7981aeb5f-model-cache\") pod \"s3-init-7sntz\" (UID: \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\") " pod="kserve/s3-init-7sntz"
Jul 13 18:57:53.700752 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.700635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vv9v\" (UniqueName: \"kubernetes.io/projected/6d6a6123-d931-4568-9ab9-14e7981aeb5f-kube-api-access-5vv9v\") pod \"s3-init-7sntz\" (UID: \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\") " pod="kserve/s3-init-7sntz"
Jul 13 18:57:53.801921 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.801893    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d6a6123-d931-4568-9ab9-14e7981aeb5f-model-cache\") pod \"s3-init-7sntz\" (UID: \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\") " pod="kserve/s3-init-7sntz"
Jul 13 18:57:53.801921 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.801921    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vv9v\" (UniqueName: \"kubernetes.io/projected/6d6a6123-d931-4568-9ab9-14e7981aeb5f-kube-api-access-5vv9v\") pod \"s3-init-7sntz\" (UID: \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\") " pod="kserve/s3-init-7sntz"
Jul 13 18:57:53.802279 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.802261    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d6a6123-d931-4568-9ab9-14e7981aeb5f-model-cache\") pod \"s3-init-7sntz\" (UID: \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\") " pod="kserve/s3-init-7sntz"
Jul 13 18:57:53.810623 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.810599    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vv9v\" (UniqueName: \"kubernetes.io/projected/6d6a6123-d931-4568-9ab9-14e7981aeb5f-kube-api-access-5vv9v\") pod \"s3-init-7sntz\" (UID: \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\") " pod="kserve/s3-init-7sntz"
Jul 13 18:57:53.931042 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:53.931000    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7sntz"
Jul 13 18:57:54.057762 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:54.057586    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-7sntz"]
Jul 13 18:57:54.060419 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:57:54.060389    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d6a6123_d931_4568_9ab9_14e7981aeb5f.slice/crio-792900db3e9f0da9e53000fe1555ccdf35141a4b68663481d716c0b283c03a20 WatchSource:0}: Error finding container 792900db3e9f0da9e53000fe1555ccdf35141a4b68663481d716c0b283c03a20: Status 404 returned error can't find the container with id 792900db3e9f0da9e53000fe1555ccdf35141a4b68663481d716c0b283c03a20
Jul 13 18:57:54.871437 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:54.871400    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7sntz" event={"ID":"6d6a6123-d931-4568-9ab9-14e7981aeb5f","Type":"ContainerStarted","Data":"792900db3e9f0da9e53000fe1555ccdf35141a4b68663481d716c0b283c03a20"}
Jul 13 18:57:58.888341 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:58.888307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7sntz" event={"ID":"6d6a6123-d931-4568-9ab9-14e7981aeb5f","Type":"ContainerStarted","Data":"85ee316bdf656e08728785aad69d79761ac438016d1b74bac75a50abd28cb693"}
Jul 13 18:57:59.893508 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:59.893472    2569 generic.go:356] "Generic (PLEG): container finished" podID="6d6a6123-d931-4568-9ab9-14e7981aeb5f" containerID="85ee316bdf656e08728785aad69d79761ac438016d1b74bac75a50abd28cb693" exitCode=0
Jul 13 18:57:59.893930 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:57:59.893516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7sntz" event={"ID":"6d6a6123-d931-4568-9ab9-14e7981aeb5f","Type":"ContainerDied","Data":"85ee316bdf656e08728785aad69d79761ac438016d1b74bac75a50abd28cb693"}
Jul 13 18:58:04.915489 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:58:04.915447    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7sntz" event={"ID":"6d6a6123-d931-4568-9ab9-14e7981aeb5f","Type":"ContainerStarted","Data":"fe8768d5efb532cbf056c393e81468807f6511d0593ac267d1993aa04d4409ae"}
Jul 13 18:58:04.933256 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:58:04.933200    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-7sntz" podStartSLOduration=1.711988491 podStartE2EDuration="11.933186156s" totalImagesPullingTime="8.942324715s" totalInitContainerRuntime="1.27887295s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:57:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:58:04.931504767 +0000 UTC m=+477.037108387" watchObservedRunningTime="2026-07-13 18:58:04.933186156 +0000 UTC m=+477.038789775"
Jul 13 18:58:53.787839 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:58:53.787787    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:10.145947 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:10.145912    2569 generic.go:356] "Generic (PLEG): container finished" podID="6d6a6123-d931-4568-9ab9-14e7981aeb5f" containerID="fe8768d5efb532cbf056c393e81468807f6511d0593ac267d1993aa04d4409ae" exitCode=0
Jul 13 18:59:10.146365 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:10.145993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7sntz" event={"ID":"6d6a6123-d931-4568-9ab9-14e7981aeb5f","Type":"ContainerDied","Data":"fe8768d5efb532cbf056c393e81468807f6511d0593ac267d1993aa04d4409ae"}
Jul 13 18:59:11.271907 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:11.271885    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7sntz"
Jul 13 18:59:11.422021 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:11.421936    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d6a6123-d931-4568-9ab9-14e7981aeb5f-model-cache\") pod \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\" (UID: \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\") "
Jul 13 18:59:11.422021 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:11.422003    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vv9v\" (UniqueName: \"kubernetes.io/projected/6d6a6123-d931-4568-9ab9-14e7981aeb5f-kube-api-access-5vv9v\") pod \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\" (UID: \"6d6a6123-d931-4568-9ab9-14e7981aeb5f\") "
Jul 13 18:59:11.424164 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:11.424133    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6a6123-d931-4568-9ab9-14e7981aeb5f-kube-api-access-5vv9v" (OuterVolumeSpecName: "kube-api-access-5vv9v") pod "6d6a6123-d931-4568-9ab9-14e7981aeb5f" (UID: "6d6a6123-d931-4568-9ab9-14e7981aeb5f"). InnerVolumeSpecName "kube-api-access-5vv9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:59:11.440782 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:11.440744    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6a6123-d931-4568-9ab9-14e7981aeb5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d6a6123-d931-4568-9ab9-14e7981aeb5f" (UID: "6d6a6123-d931-4568-9ab9-14e7981aeb5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:59:11.522774 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:11.522738    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d6a6123-d931-4568-9ab9-14e7981aeb5f-model-cache\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:59:11.522774 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:11.522768    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vv9v\" (UniqueName: \"kubernetes.io/projected/6d6a6123-d931-4568-9ab9-14e7981aeb5f-kube-api-access-5vv9v\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 18:59:12.154612 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:12.154584    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-7sntz"
Jul 13 18:59:12.154754 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:12.154617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-7sntz" event={"ID":"6d6a6123-d931-4568-9ab9-14e7981aeb5f","Type":"ContainerDied","Data":"792900db3e9f0da9e53000fe1555ccdf35141a4b68663481d716c0b283c03a20"}
Jul 13 18:59:12.154754 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:12.154648    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="792900db3e9f0da9e53000fe1555ccdf35141a4b68663481d716c0b283c03a20"
Jul 13 18:59:12.560084 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:12.560052    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:12.764368 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:12.764336    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:23.941087 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.941052    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:23.944246 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.944214    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"]
Jul 13 18:59:23.944626 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.944595    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6a6123-d931-4568-9ab9-14e7981aeb5f" containerName="s3-init"
Jul 13 18:59:23.944626 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.944617    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6a6123-d931-4568-9ab9-14e7981aeb5f" containerName="s3-init"
Jul 13 18:59:23.944805 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.944637    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6a6123-d931-4568-9ab9-14e7981aeb5f" containerName="download-hf-model"
Jul 13 18:59:23.944805 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.944647    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6a6123-d931-4568-9ab9-14e7981aeb5f" containerName="download-hf-model"
Jul 13 18:59:23.944805 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.944715    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d6a6123-d931-4568-9ab9-14e7981aeb5f" containerName="s3-init"
Jul 13 18:59:23.947317 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.947300    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:23.950014 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.949991    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 18:59:23.950152 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.950025    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 18:59:23.950152 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.950036    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 18:59:23.950152 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.950042    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-9e27e-predictor-serving-cert\""
Jul 13 18:59:23.950152 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.950057    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mgp78\""
Jul 13 18:59:23.950152 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.949993    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-9e27e-kube-rbac-proxy-sar-config\""
Jul 13 18:59:23.950402 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.950374    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 18:59:23.959202 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:23.959180    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"]
Jul 13 18:59:24.015815 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.015746    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97b35062-22c9-418f-854a-24590b93238c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.015981 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.015875    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.015981 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.015917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b35062-22c9-418f-854a-24590b93238c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.015981 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.015956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.016096 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.015977    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvc2\" (UniqueName: \"kubernetes.io/projected/97b35062-22c9-418f-854a-24590b93238c-kube-api-access-jwvc2\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.117192 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.117151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.117377 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.117210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b35062-22c9-418f-854a-24590b93238c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.117377 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.117235    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.117377 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.117262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvc2\" (UniqueName: \"kubernetes.io/projected/97b35062-22c9-418f-854a-24590b93238c-kube-api-access-jwvc2\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.117377 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.117326    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97b35062-22c9-418f-854a-24590b93238c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.117581 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:59:24.117387    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-9e27e-predictor-serving-cert" not found
Jul 13 18:59:24.117581 ip-10-0-130-96 kubenswrapper[2569]: E0713 18:59:24.117467    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b35062-22c9-418f-854a-24590b93238c-proxy-tls podName:97b35062-22c9-418f-854a-24590b93238c nodeName:}" failed. No retries permitted until 2026-07-13 18:59:24.617450412 +0000 UTC m=+556.723054011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/97b35062-22c9-418f-854a-24590b93238c-proxy-tls") pod "isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" (UID: "97b35062-22c9-418f-854a-24590b93238c") : secret "isvc-raw-sklearn-batcher-9e27e-predictor-serving-cert" not found
Jul 13 18:59:24.117735 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.117711    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97b35062-22c9-418f-854a-24590b93238c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.117958 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.117941    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.118006 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.117990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.126428 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.126402    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvc2\" (UniqueName: \"kubernetes.io/projected/97b35062-22c9-418f-854a-24590b93238c-kube-api-access-jwvc2\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.259555 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.259459    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 18:59:24.621742 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.621655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b35062-22c9-418f-854a-24590b93238c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.624044 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.624015    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b35062-22c9-418f-854a-24590b93238c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.857366 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.857328    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:24.986915 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:24.986884    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"]
Jul 13 18:59:24.989330 ip-10-0-130-96 kubenswrapper[2569]: W0713 18:59:24.989302    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b35062_22c9_418f_854a_24590b93238c.slice/crio-13a20070c7198562acb37a2894e502025d11770790e8017096e95165fe53f51f WatchSource:0}: Error finding container 13a20070c7198562acb37a2894e502025d11770790e8017096e95165fe53f51f: Status 404 returned error can't find the container with id 13a20070c7198562acb37a2894e502025d11770790e8017096e95165fe53f51f
Jul 13 18:59:25.205356 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:25.205271    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerStarted","Data":"d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d"}
Jul 13 18:59:25.205356 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:25.205342    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerStarted","Data":"13a20070c7198562acb37a2894e502025d11770790e8017096e95165fe53f51f"}
Jul 13 18:59:29.222132 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:29.222094    2569 generic.go:356] "Generic (PLEG): container finished" podID="97b35062-22c9-418f-854a-24590b93238c" containerID="d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d" exitCode=0
Jul 13 18:59:29.222578 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:29.222170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerDied","Data":"d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d"}
Jul 13 18:59:43.285270 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:43.285234    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerStarted","Data":"02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5"}
Jul 13 18:59:46.299457 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:46.299376    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerStarted","Data":"4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9"}
Jul 13 18:59:49.313259 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:49.313165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerStarted","Data":"1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a"}
Jul 13 18:59:49.313706 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:49.313332    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:50.316993 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:50.316960    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:50.317373 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:50.317036    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:50.318582 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:50.318509    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 18:59:50.319303 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:50.319274    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 18:59:50.322117 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:50.322097    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 18:59:50.347828 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:50.347784    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podStartSLOduration=4.015696977 podStartE2EDuration="27.347770321s" totalImagesPullingTime="19.707602232s" totalInitContainerRuntime="3.624471112s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:49.354662305 +0000 UTC m=+581.460265924" watchObservedRunningTime="2026-07-13 18:59:50.347770321 +0000 UTC m=+582.453373940"
Jul 13 18:59:51.324844 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:51.324795    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 18:59:51.325250 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:51.325109    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 18:59:52.328101 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:52.328060    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 18:59:52.328579 ip-10-0-130-96 kubenswrapper[2569]: I0713 18:59:52.328367    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:00:02.328596 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:02.328521    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 19:00:02.328994 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:02.328970    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:00:09.898571 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:09.898523    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 19:00:09.899069 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:09.898796    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 19:00:12.328211 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:12.328157    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 19:00:12.396450 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:12.328665    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:00:17.135014 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:17.134969    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:20.006240 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:20.006202    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:00:22.328914 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:22.328859    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 19:00:22.329286 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:22.329206    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:00:32.328856 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:32.328753    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 19:00:32.329314 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:32.329243    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:00:42.328639 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:42.328585    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 19:00:42.329145 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:42.328977    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:00:52.328521 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:52.328478    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 19:00:52.329024 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:00:52.328948    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:01:02.328959 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:02.328918    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 19:01:02.329441 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:02.329015    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 19:01:09.043598 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.043568    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"]
Jul 13 19:01:09.043961 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.043827    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" containerID="cri-o://02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5" gracePeriod=30
Jul 13 19:01:09.043961 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.043852    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" containerID="cri-o://1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a" gracePeriod=30
Jul 13 19:01:09.043961 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.043885    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy" containerID="cri-o://4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9" gracePeriod=30
Jul 13 19:01:09.163613 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.163585    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"]
Jul 13 19:01:09.166171 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.166148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.169186 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.169146    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c7cea-predictor-serving-cert\""
Jul 13 19:01:09.169313 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.169153    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c7cea-kube-rbac-proxy-sar-config\""
Jul 13 19:01:09.178459 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.178431    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"]
Jul 13 19:01:09.200677 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.200652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.200784 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.200692    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.200784 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.200726    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trcn\" (UniqueName: \"kubernetes.io/projected/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-api-access-2trcn\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.200784 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.200756    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d504da-e581-4fe3-b598-31c5d49d613b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.200912 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.200838    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d504da-e581-4fe3-b598-31c5d49d613b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.239047 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.239015    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"]
Jul 13 19:01:09.243277 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.243245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.246575 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.246550    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c7cea-predictor-serving-cert\""
Jul 13 19:01:09.246699 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.246557    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c7cea-kube-rbac-proxy-sar-config\""
Jul 13 19:01:09.246773 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.246746    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:09.254781 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.254758    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"]
Jul 13 19:01:09.302071 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.301992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqnf\" (UniqueName: \"kubernetes.io/projected/7f985c9a-3629-4e57-8d3e-855799390a54-kube-api-access-jcqnf\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.302071 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.302071 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f985c9a-3629-4e57-8d3e-855799390a54-proxy-tls\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.302071 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.302414 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f985c9a-3629-4e57-8d3e-855799390a54-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.302414 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.302414 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trcn\" (UniqueName: \"kubernetes.io/projected/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-api-access-2trcn\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.302414 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d504da-e581-4fe3-b598-31c5d49d613b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.302414 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.302414 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d504da-e581-4fe3-b598-31c5d49d613b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.302769 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:01:09.302445    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-serving-cert: secret "isvc-sklearn-graph-raw-c7cea-predictor-serving-cert" not found
Jul 13 19:01:09.302769 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:01:09.302524    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d504da-e581-4fe3-b598-31c5d49d613b-proxy-tls podName:f1d504da-e581-4fe3-b598-31c5d49d613b nodeName:}" failed. No retries permitted until 2026-07-13 19:01:09.802499103 +0000 UTC m=+661.908102709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f1d504da-e581-4fe3-b598-31c5d49d613b-proxy-tls") pod "isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" (UID: "f1d504da-e581-4fe3-b598-31c5d49d613b") : secret "isvc-sklearn-graph-raw-c7cea-predictor-serving-cert" not found
Jul 13 19:01:09.302769 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302551    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d504da-e581-4fe3-b598-31c5d49d613b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.302769 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302726    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.302961 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.302868    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.310940 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.310906    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trcn\" (UniqueName: \"kubernetes.io/projected/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-api-access-2trcn\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.346979 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.346954    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:09.403304 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.403272    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f985c9a-3629-4e57-8d3e-855799390a54-proxy-tls\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.403304 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.403309    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.403597 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.403333    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f985c9a-3629-4e57-8d3e-855799390a54-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.403597 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.403403    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.403597 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.403458    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqnf\" (UniqueName: \"kubernetes.io/projected/7f985c9a-3629-4e57-8d3e-855799390a54-kube-api-access-jcqnf\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.403872 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.403846    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f985c9a-3629-4e57-8d3e-855799390a54-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.404048 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.404029    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.404086 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.404074    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.405783 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.405753    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f985c9a-3629-4e57-8d3e-855799390a54-proxy-tls\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.413207 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.413181    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqnf\" (UniqueName: \"kubernetes.io/projected/7f985c9a-3629-4e57-8d3e-855799390a54-kube-api-access-jcqnf\") pod \"isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.555495 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.555407    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:09.628513 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.628479    2569 generic.go:356] "Generic (PLEG): container finished" podID="97b35062-22c9-418f-854a-24590b93238c" containerID="4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9" exitCode=2
Jul 13 19:01:09.628683 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.628560    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerDied","Data":"4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9"}
Jul 13 19:01:09.633623 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.633600    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:09.683292 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.683267    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"]
Jul 13 19:01:09.685542 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:01:09.685492    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f985c9a_3629_4e57_8d3e_855799390a54.slice/crio-6ed84b4a2ca851e43c50f96cedebf4e7b5f0ebdd4a1dbb1ff5f0de5038633be4 WatchSource:0}: Error finding container 6ed84b4a2ca851e43c50f96cedebf4e7b5f0ebdd4a1dbb1ff5f0de5038633be4: Status 404 returned error can't find the container with id 6ed84b4a2ca851e43c50f96cedebf4e7b5f0ebdd4a1dbb1ff5f0de5038633be4
Jul 13 19:01:09.808129 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.808050    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d504da-e581-4fe3-b598-31c5d49d613b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:09.810288 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:09.810265    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d504da-e581-4fe3-b598-31c5d49d613b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:10.078073 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:10.077982    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:10.191590 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:10.191560    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:10.219102 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:10.219071    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"]
Jul 13 19:01:10.221753 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:01:10.221728    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d504da_e581_4fe3_b598_31c5d49d613b.slice/crio-f5596513d21f4b485b4a3bc8c6a3f099f5c6d6d4ba5604ee465781da60b77932 WatchSource:0}: Error finding container f5596513d21f4b485b4a3bc8c6a3f099f5c6d6d4ba5604ee465781da60b77932: Status 404 returned error can't find the container with id f5596513d21f4b485b4a3bc8c6a3f099f5c6d6d4ba5604ee465781da60b77932
Jul 13 19:01:10.317911 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:10.317871    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 13 19:01:10.635142 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:10.635105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" event={"ID":"f1d504da-e581-4fe3-b598-31c5d49d613b","Type":"ContainerStarted","Data":"9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0"}
Jul 13 19:01:10.635337 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:10.635171    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" event={"ID":"f1d504da-e581-4fe3-b598-31c5d49d613b","Type":"ContainerStarted","Data":"f5596513d21f4b485b4a3bc8c6a3f099f5c6d6d4ba5604ee465781da60b77932"}
Jul 13 19:01:10.636757 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:10.636732    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" event={"ID":"7f985c9a-3629-4e57-8d3e-855799390a54","Type":"ContainerStarted","Data":"f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9"}
Jul 13 19:01:10.636897 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:10.636781    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" event={"ID":"7f985c9a-3629-4e57-8d3e-855799390a54","Type":"ContainerStarted","Data":"6ed84b4a2ca851e43c50f96cedebf4e7b5f0ebdd4a1dbb1ff5f0de5038633be4"}
Jul 13 19:01:12.328045 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:12.327993    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 19:01:12.328467 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:12.328327    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:01:13.651360 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:13.651328    2569 generic.go:356] "Generic (PLEG): container finished" podID="97b35062-22c9-418f-854a-24590b93238c" containerID="02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5" exitCode=0
Jul 13 19:01:13.651727 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:13.651395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerDied","Data":"02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5"}
Jul 13 19:01:13.652780 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:13.652756    2569 generic.go:356] "Generic (PLEG): container finished" podID="7f985c9a-3629-4e57-8d3e-855799390a54" containerID="f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9" exitCode=0
Jul 13 19:01:13.652883 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:13.652813    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" event={"ID":"7f985c9a-3629-4e57-8d3e-855799390a54","Type":"ContainerDied","Data":"f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9"}
Jul 13 19:01:13.734132 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:13.734099    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:01:14.660624 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:14.660587    2569 generic.go:356] "Generic (PLEG): container finished" podID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerID="9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0" exitCode=0
Jul 13 19:01:14.661077 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:14.660668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" event={"ID":"f1d504da-e581-4fe3-b598-31c5d49d613b","Type":"ContainerDied","Data":"9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0"}
Jul 13 19:01:15.317779 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:15.317738    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 13 19:01:16.671548 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:16.671501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" event={"ID":"f1d504da-e581-4fe3-b598-31c5d49d613b","Type":"ContainerStarted","Data":"19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009"}
Jul 13 19:01:16.671930 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:16.671574    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" event={"ID":"f1d504da-e581-4fe3-b598-31c5d49d613b","Type":"ContainerStarted","Data":"3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f"}
Jul 13 19:01:16.671930 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:16.671755    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:16.693899 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:16.693852    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podStartSLOduration=4.049667804 podStartE2EDuration="7.693837441s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.644169637s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:01:16.691878665 +0000 UTC m=+668.797482285" watchObservedRunningTime="2026-07-13 19:01:16.693837441 +0000 UTC m=+668.799441062"
Jul 13 19:01:17.680552 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:17.676703    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:17.682593 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:17.682550    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:01:18.681263 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:18.681221    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:01:20.318299 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:20.318248    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 13 19:01:20.318739 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:20.318384    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 19:01:22.328776 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:22.328730    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 19:01:22.329190 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:22.329071    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:01:23.686548 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:23.686497    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:01:23.687044 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:23.687019    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:01:25.317879 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:25.317828    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 13 19:01:30.318216 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:30.318169    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 13 19:01:32.328419 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:32.328322    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 13 19:01:32.328904 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:32.328495    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 19:01:32.328904 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:32.328748    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:01:32.328904 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:32.328873    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 19:01:33.687024 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:33.686984    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:01:34.752006 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:34.751975    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" event={"ID":"7f985c9a-3629-4e57-8d3e-855799390a54","Type":"ContainerStarted","Data":"a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92"}
Jul 13 19:01:34.752372 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:34.752034    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" event={"ID":"7f985c9a-3629-4e57-8d3e-855799390a54","Type":"ContainerStarted","Data":"ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12"}
Jul 13 19:01:34.752372 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:34.752301    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:34.752468 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:34.752438    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:34.753523 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:34.753497    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:01:34.776589 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:34.776518    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podStartSLOduration=1.558512036 podStartE2EDuration="25.776504315s" totalImagesPullingTime="20.683781135s" totalInitContainerRuntime="3.534211144s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:01:34.776466947 +0000 UTC m=+686.882070602" watchObservedRunningTime="2026-07-13 19:01:34.776504315 +0000 UTC m=+686.882107928"
Jul 13 19:01:35.317978 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:35.317935    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 13 19:01:35.756539 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:35.756500    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:01:39.723117 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.723085    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 19:01:39.773712 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.773681    2569 generic.go:356] "Generic (PLEG): container finished" podID="97b35062-22c9-418f-854a-24590b93238c" containerID="1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a" exitCode=0
Jul 13 19:01:39.773878 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.773748    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"
Jul 13 19:01:39.773878 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.773762    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerDied","Data":"1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a"}
Jul 13 19:01:39.773878 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.773820    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6" event={"ID":"97b35062-22c9-418f-854a-24590b93238c","Type":"ContainerDied","Data":"13a20070c7198562acb37a2894e502025d11770790e8017096e95165fe53f51f"}
Jul 13 19:01:39.773878 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.773853    2569 scope.go:117] "RemoveContainer" containerID="1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a"
Jul 13 19:01:39.781753 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.781731    2569 scope.go:117] "RemoveContainer" containerID="4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9"
Jul 13 19:01:39.789310 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.789290    2569 scope.go:117] "RemoveContainer" containerID="02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5"
Jul 13 19:01:39.796393 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.796377    2569 scope.go:117] "RemoveContainer" containerID="d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d"
Jul 13 19:01:39.803086 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.803069    2569 scope.go:117] "RemoveContainer" containerID="1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a"
Jul 13 19:01:39.803319 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:01:39.803301    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a\": container with ID starting with 1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a not found: ID does not exist" containerID="1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a"
Jul 13 19:01:39.803362 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.803330    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a"} err="failed to get container status \"1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a\": rpc error: code = NotFound desc = could not find container \"1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a\": container with ID starting with 1c06cb93480173c1763e9890d27039a6848088196d00d8e2f355748fbb54934a not found: ID does not exist"
Jul 13 19:01:39.803362 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.803349    2569 scope.go:117] "RemoveContainer" containerID="4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9"
Jul 13 19:01:39.803651 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:01:39.803634    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9\": container with ID starting with 4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9 not found: ID does not exist" containerID="4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9"
Jul 13 19:01:39.803701 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.803655    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9"} err="failed to get container status \"4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9\": rpc error: code = NotFound desc = could not find container \"4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9\": container with ID starting with 4c44298937383bb48144c7edb0002ad3c88e1bf59d4ebc20ac3f0acb18c5bfa9 not found: ID does not exist"
Jul 13 19:01:39.803701 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.803668    2569 scope.go:117] "RemoveContainer" containerID="02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5"
Jul 13 19:01:39.803870 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:01:39.803855    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5\": container with ID starting with 02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5 not found: ID does not exist" containerID="02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5"
Jul 13 19:01:39.803920 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.803873    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5"} err="failed to get container status \"02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5\": rpc error: code = NotFound desc = could not find container \"02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5\": container with ID starting with 02aeb29729c00b8b3db332608d90d6b393664cad46b386211ba39b8fe66bdfc5 not found: ID does not exist"
Jul 13 19:01:39.803920 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.803884    2569 scope.go:117] "RemoveContainer" containerID="d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d"
Jul 13 19:01:39.804070 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:01:39.804052    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d\": container with ID starting with d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d not found: ID does not exist" containerID="d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d"
Jul 13 19:01:39.804115 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.804076    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d"} err="failed to get container status \"d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d\": rpc error: code = NotFound desc = could not find container \"d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d\": container with ID starting with d5a993fd0155eeabc94cb274027c8efe6898cb7f83f40dbf4cb8734d2f7a660d not found: ID does not exist"
Jul 13 19:01:39.880658 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.880590    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-cabundle-cert\") pod \"97b35062-22c9-418f-854a-24590b93238c\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") "
Jul 13 19:01:39.880658 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.880642    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b35062-22c9-418f-854a-24590b93238c-proxy-tls\") pod \"97b35062-22c9-418f-854a-24590b93238c\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") "
Jul 13 19:01:39.880820 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.880672    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwvc2\" (UniqueName: \"kubernetes.io/projected/97b35062-22c9-418f-854a-24590b93238c-kube-api-access-jwvc2\") pod \"97b35062-22c9-418f-854a-24590b93238c\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") "
Jul 13 19:01:39.880820 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.880700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-kube-rbac-proxy-sar-config\") pod \"97b35062-22c9-418f-854a-24590b93238c\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") "
Jul 13 19:01:39.880820 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.880723    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97b35062-22c9-418f-854a-24590b93238c-kserve-provision-location\") pod \"97b35062-22c9-418f-854a-24590b93238c\" (UID: \"97b35062-22c9-418f-854a-24590b93238c\") "
Jul 13 19:01:39.881098 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.881075    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "97b35062-22c9-418f-854a-24590b93238c" (UID: "97b35062-22c9-418f-854a-24590b93238c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:01:39.881197 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.881155    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97b35062-22c9-418f-854a-24590b93238c" (UID: "97b35062-22c9-418f-854a-24590b93238c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:01:39.881276 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.881198    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b35062-22c9-418f-854a-24590b93238c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97b35062-22c9-418f-854a-24590b93238c" (UID: "97b35062-22c9-418f-854a-24590b93238c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:01:39.882859 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.882826    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b35062-22c9-418f-854a-24590b93238c-kube-api-access-jwvc2" (OuterVolumeSpecName: "kube-api-access-jwvc2") pod "97b35062-22c9-418f-854a-24590b93238c" (UID: "97b35062-22c9-418f-854a-24590b93238c"). InnerVolumeSpecName "kube-api-access-jwvc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:01:39.883099 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.883079    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b35062-22c9-418f-854a-24590b93238c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97b35062-22c9-418f-854a-24590b93238c" (UID: "97b35062-22c9-418f-854a-24590b93238c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:01:39.981827 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.981786    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:01:39.981827 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.981820    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97b35062-22c9-418f-854a-24590b93238c-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:01:39.982029 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.981836    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97b35062-22c9-418f-854a-24590b93238c-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:01:39.982029 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.981851    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b35062-22c9-418f-854a-24590b93238c-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:01:39.982029 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:39.981864    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwvc2\" (UniqueName: \"kubernetes.io/projected/97b35062-22c9-418f-854a-24590b93238c-kube-api-access-jwvc2\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:01:40.092910 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:40.092880    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"]
Jul 13 19:01:40.100608 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:40.100584    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9e27e-predictor-54c57f86c5-kzcm6"]
Jul 13 19:01:40.761398 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:40.761370    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:01:40.761841 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:40.761815    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:01:42.045279 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:42.045242    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97b35062-22c9-418f-854a-24590b93238c" path="/var/lib/kubelet/pods/97b35062-22c9-418f-854a-24590b93238c/volumes"
Jul 13 19:01:43.687410 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:43.687369    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:01:50.762593 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:50.762547    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:01:53.687038 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:01:53.686991    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:02:00.762842 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:00.762738    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:02:03.687507 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:03.687464    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:02:10.762226 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:10.762187    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:02:13.687512 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:13.687471    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:02:20.762094 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:20.762050    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:02:23.687640 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:23.687598    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:02:30.761914 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:30.761872    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:02:33.687692 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:33.687664    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:02:40.763234 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:40.763199    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:02:49.218869 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:49.218832    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:49.772662 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:49.772604    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:59.398164 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.398129    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:59.598287 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.598248    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"]
Jul 13 19:02:59.598606 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.598579    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" containerID="cri-o://3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f" gracePeriod=30
Jul 13 19:02:59.598729 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.598587    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kube-rbac-proxy" containerID="cri-o://19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009" gracePeriod=30
Jul 13 19:02:59.615885 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.615852    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:59.740298 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.740258    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:59.835246 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.835207    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"]
Jul 13 19:02:59.835506 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.835482    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" containerID="cri-o://ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12" gracePeriod=30
Jul 13 19:02:59.835638 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.835518    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kube-rbac-proxy" containerID="cri-o://a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92" gracePeriod=30
Jul 13 19:02:59.842082 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842060    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"]
Jul 13 19:02:59.842389 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842376    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="storage-initializer"
Jul 13 19:02:59.842428 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842391    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="storage-initializer"
Jul 13 19:02:59.842428 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842403    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy"
Jul 13 19:02:59.842428 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842409    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy"
Jul 13 19:02:59.842428 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842423    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent"
Jul 13 19:02:59.842428 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842429    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent"
Jul 13 19:02:59.842599 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842441    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container"
Jul 13 19:02:59.842599 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842446    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container"
Jul 13 19:02:59.842599 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842492    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kube-rbac-proxy"
Jul 13 19:02:59.842599 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842499    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="agent"
Jul 13 19:02:59.842599 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.842508    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b35062-22c9-418f-854a-24590b93238c" containerName="kserve-container"
Jul 13 19:02:59.844909 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.844893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:02:59.847484 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.847465    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-be515-predictor-serving-cert\""
Jul 13 19:02:59.850371 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.850352    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-be515-kube-rbac-proxy-sar-config\""
Jul 13 19:02:59.862453 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.862435    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"]
Jul 13 19:02:59.896460 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.896434    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:02:59.916066 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.916041    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:02:59.916178 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.916074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fc098c6-da5b-4d00-a45d-9e083e7d8406-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:02:59.916178 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.916095    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:02:59.916178 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.916115    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wtf\" (UniqueName: \"kubernetes.io/projected/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-api-access-b6wtf\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:02:59.916283 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.916212    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:02:59.957487 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.957463    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"]
Jul 13 19:02:59.959954 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.959912    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:02:59.963019 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.963000    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-be515-predictor-serving-cert\""
Jul 13 19:02:59.963498 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.963481    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-be515-kube-rbac-proxy-sar-config\""
Jul 13 19:02:59.977031 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:02:59.977011    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"]
Jul 13 19:03:00.005077 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.005054    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:00.016969 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.016945    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.017067 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.016980    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e07a79b0-93de-4baf-af00-5cfb9a30a665-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.017067 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.017067 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvmf\" (UniqueName: \"kubernetes.io/projected/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-api-access-bwvmf\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.017067 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.017257 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fc098c6-da5b-4d00-a45d-9e083e7d8406-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.017257 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.017257 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wtf\" (UniqueName: \"kubernetes.io/projected/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-api-access-b6wtf\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.017257 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017156    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.017257 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017212    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e07a79b0-93de-4baf-af00-5cfb9a30a665-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.017482 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017468    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.017764 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017741    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.017847 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.017741    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.019542 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.019515    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fc098c6-da5b-4d00-a45d-9e083e7d8406-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.034943 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.034925    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wtf\" (UniqueName: \"kubernetes.io/projected/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-api-access-b6wtf\") pod \"isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.083811 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.083780    2569 generic.go:356] "Generic (PLEG): container finished" podID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerID="19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009" exitCode=2
Jul 13 19:03:00.083955 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.083870    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" event={"ID":"f1d504da-e581-4fe3-b598-31c5d49d613b","Type":"ContainerDied","Data":"19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009"}
Jul 13 19:03:00.086138 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.086108    2569 generic.go:356] "Generic (PLEG): container finished" podID="7f985c9a-3629-4e57-8d3e-855799390a54" containerID="a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92" exitCode=2
Jul 13 19:03:00.086267 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.086137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" event={"ID":"7f985c9a-3629-4e57-8d3e-855799390a54","Type":"ContainerDied","Data":"a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92"}
Jul 13 19:03:00.102901 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.102878    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:00.117796 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.117779    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e07a79b0-93de-4baf-af00-5cfb9a30a665-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.117888 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.117820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.117888 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.117839    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvmf\" (UniqueName: \"kubernetes.io/projected/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-api-access-bwvmf\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.117982 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.117881    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.117982 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.117914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e07a79b0-93de-4baf-af00-5cfb9a30a665-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.118108 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:03:00.118092    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-be515-predictor-serving-cert" not found
Jul 13 19:03:00.118165 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.118148    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e07a79b0-93de-4baf-af00-5cfb9a30a665-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.118165 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:03:00.118158    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e07a79b0-93de-4baf-af00-5cfb9a30a665-proxy-tls podName:e07a79b0-93de-4baf-af00-5cfb9a30a665 nodeName:}" failed. No retries permitted until 2026-07-13 19:03:00.618139105 +0000 UTC m=+772.723742702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e07a79b0-93de-4baf-af00-5cfb9a30a665-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" (UID: "e07a79b0-93de-4baf-af00-5cfb9a30a665") : secret "isvc-xgboost-graph-raw-hpa-be515-predictor-serving-cert" not found
Jul 13 19:03:00.118448 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.118430    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.118583 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.118565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.129017 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.128998    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvmf\" (UniqueName: \"kubernetes.io/projected/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-api-access-bwvmf\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.156673 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.156650    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:00.275036 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.275008    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:03:00.293854 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.289585    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"]
Jul 13 19:03:00.297705 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:03:00.297643    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc098c6_da5b_4d00_a45d_9e083e7d8406.slice/crio-f6d59c699af7b5894bd40b748b57808b8d77e37cb926f2b9e6223bbf53156eb0 WatchSource:0}: Error finding container f6d59c699af7b5894bd40b748b57808b8d77e37cb926f2b9e6223bbf53156eb0: Status 404 returned error can't find the container with id f6d59c699af7b5894bd40b748b57808b8d77e37cb926f2b9e6223bbf53156eb0
Jul 13 19:03:00.623562 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.623467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e07a79b0-93de-4baf-af00-5cfb9a30a665-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.625800 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.625777    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e07a79b0-93de-4baf-af00-5cfb9a30a665-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:00.757138 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.757099    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 13 19:03:00.762484 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.762451    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 13 19:03:00.869493 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:00.869456    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:01.003229 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:01.003200    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"]
Jul 13 19:03:01.004962 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:03:01.004928    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07a79b0_93de_4baf_af00_5cfb9a30a665.slice/crio-00215bedc6eb6514f3c51ed2ae6efe680c63177ac0c2fe876d9861ce02af2c0b WatchSource:0}: Error finding container 00215bedc6eb6514f3c51ed2ae6efe680c63177ac0c2fe876d9861ce02af2c0b: Status 404 returned error can't find the container with id 00215bedc6eb6514f3c51ed2ae6efe680c63177ac0c2fe876d9861ce02af2c0b
Jul 13 19:03:01.091707 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:01.091673    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" event={"ID":"9fc098c6-da5b-4d00-a45d-9e083e7d8406","Type":"ContainerStarted","Data":"6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625"}
Jul 13 19:03:01.091851 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:01.091741    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" event={"ID":"9fc098c6-da5b-4d00-a45d-9e083e7d8406","Type":"ContainerStarted","Data":"f6d59c699af7b5894bd40b748b57808b8d77e37cb926f2b9e6223bbf53156eb0"}
Jul 13 19:03:01.093450 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:01.093424    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" event={"ID":"e07a79b0-93de-4baf-af00-5cfb9a30a665","Type":"ContainerStarted","Data":"8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7"}
Jul 13 19:03:01.093576 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:01.093475    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" event={"ID":"e07a79b0-93de-4baf-af00-5cfb9a30a665","Type":"ContainerStarted","Data":"00215bedc6eb6514f3c51ed2ae6efe680c63177ac0c2fe876d9861ce02af2c0b"}
Jul 13 19:03:03.681426 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.681389    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 13 19:03:03.682952 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.682933    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:03:03.687677 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.687651    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 13 19:03:03.752667 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.752586    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f985c9a-3629-4e57-8d3e-855799390a54-proxy-tls\") pod \"7f985c9a-3629-4e57-8d3e-855799390a54\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") "
Jul 13 19:03:03.752667 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.752625    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-kube-rbac-proxy-sar-config\") pod \"7f985c9a-3629-4e57-8d3e-855799390a54\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") "
Jul 13 19:03:03.752878 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.752673    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f985c9a-3629-4e57-8d3e-855799390a54-kserve-provision-location\") pod \"7f985c9a-3629-4e57-8d3e-855799390a54\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") "
Jul 13 19:03:03.752878 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.752712    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcqnf\" (UniqueName: \"kubernetes.io/projected/7f985c9a-3629-4e57-8d3e-855799390a54-kube-api-access-jcqnf\") pod \"7f985c9a-3629-4e57-8d3e-855799390a54\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") "
Jul 13 19:03:03.752878 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.752728    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-cabundle-cert\") pod \"7f985c9a-3629-4e57-8d3e-855799390a54\" (UID: \"7f985c9a-3629-4e57-8d3e-855799390a54\") "
Jul 13 19:03:03.753062 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.752994    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f985c9a-3629-4e57-8d3e-855799390a54-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f985c9a-3629-4e57-8d3e-855799390a54" (UID: "7f985c9a-3629-4e57-8d3e-855799390a54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:03:03.753124 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.753091    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f985c9a-3629-4e57-8d3e-855799390a54" (UID: "7f985c9a-3629-4e57-8d3e-855799390a54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:03:03.753214 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.753193    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f985c9a-3629-4e57-8d3e-855799390a54" (UID: "7f985c9a-3629-4e57-8d3e-855799390a54"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:03:03.754829 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.754803    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f985c9a-3629-4e57-8d3e-855799390a54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f985c9a-3629-4e57-8d3e-855799390a54" (UID: "7f985c9a-3629-4e57-8d3e-855799390a54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:03:03.754916 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.754892    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f985c9a-3629-4e57-8d3e-855799390a54-kube-api-access-jcqnf" (OuterVolumeSpecName: "kube-api-access-jcqnf") pod "7f985c9a-3629-4e57-8d3e-855799390a54" (UID: "7f985c9a-3629-4e57-8d3e-855799390a54"). InnerVolumeSpecName "kube-api-access-jcqnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:03:03.853885 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.853843    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f985c9a-3629-4e57-8d3e-855799390a54-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:03.853885 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.853876    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:03.853885 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.853886    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f985c9a-3629-4e57-8d3e-855799390a54-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:03.853885 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.853896    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcqnf\" (UniqueName: \"kubernetes.io/projected/7f985c9a-3629-4e57-8d3e-855799390a54-kube-api-access-jcqnf\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:03.854181 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:03.853906    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f985c9a-3629-4e57-8d3e-855799390a54-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:04.107003 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.106958    2569 generic.go:356] "Generic (PLEG): container finished" podID="7f985c9a-3629-4e57-8d3e-855799390a54" containerID="ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12" exitCode=0
Jul 13 19:03:04.107148 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.107020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" event={"ID":"7f985c9a-3629-4e57-8d3e-855799390a54","Type":"ContainerDied","Data":"ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12"}
Jul 13 19:03:04.107148 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.107034    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"
Jul 13 19:03:04.107148 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.107105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4" event={"ID":"7f985c9a-3629-4e57-8d3e-855799390a54","Type":"ContainerDied","Data":"6ed84b4a2ca851e43c50f96cedebf4e7b5f0ebdd4a1dbb1ff5f0de5038633be4"}
Jul 13 19:03:04.107148 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.107143    2569 scope.go:117] "RemoveContainer" containerID="a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92"
Jul 13 19:03:04.108695 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.108669    2569 generic.go:356] "Generic (PLEG): container finished" podID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerID="6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625" exitCode=0
Jul 13 19:03:04.108842 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.108705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" event={"ID":"9fc098c6-da5b-4d00-a45d-9e083e7d8406","Type":"ContainerDied","Data":"6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625"}
Jul 13 19:03:04.115734 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.115719    2569 scope.go:117] "RemoveContainer" containerID="ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12"
Jul 13 19:03:04.123122 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.123108    2569 scope.go:117] "RemoveContainer" containerID="f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9"
Jul 13 19:03:04.128764 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.128745    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"]
Jul 13 19:03:04.132372 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.132355    2569 scope.go:117] "RemoveContainer" containerID="a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92"
Jul 13 19:03:04.132711 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:03:04.132680    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92\": container with ID starting with a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92 not found: ID does not exist" containerID="a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92"
Jul 13 19:03:04.132794 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.132718    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92"} err="failed to get container status \"a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92\": rpc error: code = NotFound desc = could not find container \"a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92\": container with ID starting with a89a546913610651a78067b21942c8a9f64916cc9d86df48bb3cc22a1b874e92 not found: ID does not exist"
Jul 13 19:03:04.132794 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.132735    2569 scope.go:117] "RemoveContainer" containerID="ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12"
Jul 13 19:03:04.133147 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:03:04.133023    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12\": container with ID starting with ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12 not found: ID does not exist" containerID="ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12"
Jul 13 19:03:04.133147 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.133084    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12"} err="failed to get container status \"ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12\": rpc error: code = NotFound desc = could not find container \"ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12\": container with ID starting with ddf1f02d0429162a0cd37b69d476b69198f331e2dfb8c6bb2fdb38141e8f2c12 not found: ID does not exist"
Jul 13 19:03:04.133147 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.133109    2569 scope.go:117] "RemoveContainer" containerID="f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9"
Jul 13 19:03:04.133597 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:03:04.133571    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9\": container with ID starting with f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9 not found: ID does not exist" containerID="f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9"
Jul 13 19:03:04.133727 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.133605    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9"} err="failed to get container status \"f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9\": rpc error: code = NotFound desc = could not find container \"f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9\": container with ID starting with f089df2cdd3e80217d7e4a228846e531156344cb7dcd048a687a464a888c29d9 not found: ID does not exist"
Jul 13 19:03:04.135203 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.135181    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c7cea-predictor-86bcb78845-4h9q4"]
Jul 13 19:03:04.423508 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.423478    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:03:04.558910 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.558877    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-rbac-proxy-sar-config\") pod \"f1d504da-e581-4fe3-b598-31c5d49d613b\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") "
Jul 13 19:03:04.558910 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.558917    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d504da-e581-4fe3-b598-31c5d49d613b-kserve-provision-location\") pod \"f1d504da-e581-4fe3-b598-31c5d49d613b\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") "
Jul 13 19:03:04.559314 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.558955    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-cabundle-cert\") pod \"f1d504da-e581-4fe3-b598-31c5d49d613b\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") "
Jul 13 19:03:04.559314 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.559041    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d504da-e581-4fe3-b598-31c5d49d613b-proxy-tls\") pod \"f1d504da-e581-4fe3-b598-31c5d49d613b\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") "
Jul 13 19:03:04.559314 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.559074    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2trcn\" (UniqueName: \"kubernetes.io/projected/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-api-access-2trcn\") pod \"f1d504da-e581-4fe3-b598-31c5d49d613b\" (UID: \"f1d504da-e581-4fe3-b598-31c5d49d613b\") "
Jul 13 19:03:04.559314 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.559284    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f1d504da-e581-4fe3-b598-31c5d49d613b" (UID: "f1d504da-e581-4fe3-b598-31c5d49d613b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:03:04.559520 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.559332    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d504da-e581-4fe3-b598-31c5d49d613b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1d504da-e581-4fe3-b598-31c5d49d613b" (UID: "f1d504da-e581-4fe3-b598-31c5d49d613b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:03:04.559520 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.559448    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1d504da-e581-4fe3-b598-31c5d49d613b" (UID: "f1d504da-e581-4fe3-b598-31c5d49d613b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:03:04.561148 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.561121    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-api-access-2trcn" (OuterVolumeSpecName: "kube-api-access-2trcn") pod "f1d504da-e581-4fe3-b598-31c5d49d613b" (UID: "f1d504da-e581-4fe3-b598-31c5d49d613b"). InnerVolumeSpecName "kube-api-access-2trcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:03:04.561219 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.561176    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d504da-e581-4fe3-b598-31c5d49d613b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1d504da-e581-4fe3-b598-31c5d49d613b" (UID: "f1d504da-e581-4fe3-b598-31c5d49d613b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:03:04.659667 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.659632    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2trcn\" (UniqueName: \"kubernetes.io/projected/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-api-access-2trcn\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:04.659667 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.659660    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:04.659667 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.659673    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1d504da-e581-4fe3-b598-31c5d49d613b-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:04.659893 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.659685    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1d504da-e581-4fe3-b598-31c5d49d613b-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:04.659893 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:04.659697    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1d504da-e581-4fe3-b598-31c5d49d613b-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:03:05.114157 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.114077    2569 generic.go:356] "Generic (PLEG): container finished" podID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerID="8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7" exitCode=0
Jul 13 19:03:05.114637 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.114149    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" event={"ID":"e07a79b0-93de-4baf-af00-5cfb9a30a665","Type":"ContainerDied","Data":"8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7"}
Jul 13 19:03:05.116135 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.116100    2569 generic.go:356] "Generic (PLEG): container finished" podID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerID="3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f" exitCode=0
Jul 13 19:03:05.116273 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.116147    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"
Jul 13 19:03:05.116273 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.116157    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" event={"ID":"f1d504da-e581-4fe3-b598-31c5d49d613b","Type":"ContainerDied","Data":"3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f"}
Jul 13 19:03:05.116273 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.116216    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5" event={"ID":"f1d504da-e581-4fe3-b598-31c5d49d613b","Type":"ContainerDied","Data":"f5596513d21f4b485b4a3bc8c6a3f099f5c6d6d4ba5604ee465781da60b77932"}
Jul 13 19:03:05.116273 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.116254    2569 scope.go:117] "RemoveContainer" containerID="19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009"
Jul 13 19:03:05.119627 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.119603    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" event={"ID":"9fc098c6-da5b-4d00-a45d-9e083e7d8406","Type":"ContainerStarted","Data":"50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a"}
Jul 13 19:03:05.119744 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.119659    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" event={"ID":"9fc098c6-da5b-4d00-a45d-9e083e7d8406","Type":"ContainerStarted","Data":"f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4"}
Jul 13 19:03:05.119888 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.119872    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:05.126237 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.126220    2569 scope.go:117] "RemoveContainer" containerID="3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f"
Jul 13 19:03:05.139761 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.139741    2569 scope.go:117] "RemoveContainer" containerID="9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0"
Jul 13 19:03:05.153663 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.153643    2569 scope.go:117] "RemoveContainer" containerID="19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009"
Jul 13 19:03:05.153946 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:03:05.153929    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009\": container with ID starting with 19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009 not found: ID does not exist" containerID="19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009"
Jul 13 19:03:05.153990 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.153954    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009"} err="failed to get container status \"19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009\": rpc error: code = NotFound desc = could not find container \"19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009\": container with ID starting with 19b4e50ac5c15bb413404253c3e88eb94e3b293b516fb9660691bc132918e009 not found: ID does not exist"
Jul 13 19:03:05.153990 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.153970    2569 scope.go:117] "RemoveContainer" containerID="3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f"
Jul 13 19:03:05.154208 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:03:05.154187    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f\": container with ID starting with 3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f not found: ID does not exist" containerID="3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f"
Jul 13 19:03:05.154264 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.154214    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f"} err="failed to get container status \"3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f\": rpc error: code = NotFound desc = could not find container \"3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f\": container with ID starting with 3bf8e99f55c6c33c342ac991d92f1ccd2b9339f265b44b85879baba3ece8fe0f not found: ID does not exist"
Jul 13 19:03:05.154264 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.154235    2569 scope.go:117] "RemoveContainer" containerID="9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0"
Jul 13 19:03:05.154515 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:03:05.154493    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0\": container with ID starting with 9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0 not found: ID does not exist" containerID="9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0"
Jul 13 19:03:05.154601 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.154521    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0"} err="failed to get container status \"9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0\": rpc error: code = NotFound desc = could not find container \"9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0\": container with ID starting with 9a3b24ceef5f2a98c79c82cc7cf975a131be7256854bda59ba29e48b957977d0 not found: ID does not exist"
Jul 13 19:03:05.162582 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.162523    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podStartSLOduration=2.652872078 podStartE2EDuration="6.162511877s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.509639799s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:03:05.161144692 +0000 UTC m=+777.266748316" watchObservedRunningTime="2026-07-13 19:03:05.162511877 +0000 UTC m=+777.268115495"
Jul 13 19:03:05.176030 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.176003    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"]
Jul 13 19:03:05.180268 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:05.180240    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c7cea-predictor-8659c6db47-zk6l5"]
Jul 13 19:03:06.046494 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:06.046461    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" path="/var/lib/kubelet/pods/7f985c9a-3629-4e57-8d3e-855799390a54/volumes"
Jul 13 19:03:06.047019 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:06.047003    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" path="/var/lib/kubelet/pods/f1d504da-e581-4fe3-b598-31c5d49d613b/volumes"
Jul 13 19:03:06.125391 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:06.125352    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" event={"ID":"e07a79b0-93de-4baf-af00-5cfb9a30a665","Type":"ContainerStarted","Data":"df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e"}
Jul 13 19:03:06.125840 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:06.125427    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" event={"ID":"e07a79b0-93de-4baf-af00-5cfb9a30a665","Type":"ContainerStarted","Data":"87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d"}
Jul 13 19:03:06.125840 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:06.125738    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:06.126490 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:06.126468    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:06.127746 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:06.127719    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:03:06.153324 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:06.153283    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podStartSLOduration=3.389837033 podStartE2EDuration="7.153269609s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.763432576s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:03:06.150507518 +0000 UTC m=+778.256111160" watchObservedRunningTime="2026-07-13 19:03:06.153269609 +0000 UTC m=+778.258873228"
Jul 13 19:03:07.135334 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:07.135285    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:07.135764 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:07.135441    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:03:07.136396 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:07.136221    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:03:08.137049 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:08.137013    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:03:12.140041 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:12.140011    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:03:12.140692 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:12.140663    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:03:13.141487 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:13.141455    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:03:13.141981 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:13.141953    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:03:22.140666 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:22.140626    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:03:23.141908 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:23.141871    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:03:32.141401 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:32.141320    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:03:33.141908 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:33.141870    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:03:42.141177 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:42.141135    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:03:43.143464 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:43.143407    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:03:52.140827 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:52.140790    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:03:53.142511 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:03:53.142470    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:04:02.141012 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:02.140969    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:04:03.142383 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:03.142346    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:04:09.717976 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:09.717940    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:09.819717 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:09.819681    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:12.140603 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:12.140566    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:04:13.143711 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:13.143680    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:04:17.041414 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:17.041371    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 13 19:04:27.042492 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:27.042458    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:04:39.667480 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:39.667443    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:40.006734 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:40.006644    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:49.740889 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.740858    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:49.910957 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.910921    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:49.922637 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.922603    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"]
Jul 13 19:04:49.922921 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.922879    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container" containerID="cri-o://f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4" gracePeriod=30
Jul 13 19:04:49.923029 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.922907    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kube-rbac-proxy" containerID="cri-o://50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a" gracePeriod=30
Jul 13 19:04:49.960935 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.960902    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"]
Jul 13 19:04:49.961204 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961193    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kube-rbac-proxy"
Jul 13 19:04:49.961253 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961206    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kube-rbac-proxy"
Jul 13 19:04:49.961253 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961223    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container"
Jul 13 19:04:49.961253 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961228    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container"
Jul 13 19:04:49.961253 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961237    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container"
Jul 13 19:04:49.961253 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961243    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container"
Jul 13 19:04:49.961253 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961250    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kube-rbac-proxy"
Jul 13 19:04:49.961425 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961257    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kube-rbac-proxy"
Jul 13 19:04:49.961425 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961271    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="storage-initializer"
Jul 13 19:04:49.961425 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961279    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="storage-initializer"
Jul 13 19:04:49.961425 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961296    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="storage-initializer"
Jul 13 19:04:49.961425 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961301    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="storage-initializer"
Jul 13 19:04:49.961425 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961344    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kserve-container"
Jul 13 19:04:49.961425 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961355    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f985c9a-3629-4e57-8d3e-855799390a54" containerName="kube-rbac-proxy"
Jul 13 19:04:49.961425 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961361    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kube-rbac-proxy"
Jul 13 19:04:49.961425 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.961367    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1d504da-e581-4fe3-b598-31c5d49d613b" containerName="kserve-container"
Jul 13 19:04:49.964332 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.964314    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:49.966953 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.966936    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-2a04c-predictor-serving-cert\""
Jul 13 19:04:49.967427 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.967412    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-2a04c-kube-rbac-proxy-sar-config\""
Jul 13 19:04:49.980224 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:49.980197    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"]
Jul 13 19:04:50.002634 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.002568    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:50.018777 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.018745    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790fb00a-b1d5-4e14-8186-8e85874a0fdf-proxy-tls\") pod \"message-dumper-raw-2a04c-predictor-6bfbf69478-hm649\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") " pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.018934 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.018907    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q9h5\" (UniqueName: \"kubernetes.io/projected/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-api-access-5q9h5\") pod \"message-dumper-raw-2a04c-predictor-6bfbf69478-hm649\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") " pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.018991 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.018954    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-2a04c-predictor-6bfbf69478-hm649\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") " pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.046199 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.046174    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"]
Jul 13 19:04:50.046423 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.046383    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" containerID="cri-o://87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d" gracePeriod=30
Jul 13 19:04:50.046487 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.046403    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kube-rbac-proxy" containerID="cri-o://df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e" gracePeriod=30
Jul 13 19:04:50.086797 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.086759    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:50.119378 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.119342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q9h5\" (UniqueName: \"kubernetes.io/projected/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-api-access-5q9h5\") pod \"message-dumper-raw-2a04c-predictor-6bfbf69478-hm649\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") " pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.119378 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.119378    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-2a04c-predictor-6bfbf69478-hm649\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") " pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.119653 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.119413    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790fb00a-b1d5-4e14-8186-8e85874a0fdf-proxy-tls\") pod \"message-dumper-raw-2a04c-predictor-6bfbf69478-hm649\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") " pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.120139 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.120114    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-2a04c-predictor-6bfbf69478-hm649\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") " pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.122277 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.122253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790fb00a-b1d5-4e14-8186-8e85874a0fdf-proxy-tls\") pod \"message-dumper-raw-2a04c-predictor-6bfbf69478-hm649\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") " pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.128436 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.128402    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q9h5\" (UniqueName: \"kubernetes.io/projected/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-api-access-5q9h5\") pod \"message-dumper-raw-2a04c-predictor-6bfbf69478-hm649\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") " pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.274146 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.274067    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:50.405308 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.405283    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"]
Jul 13 19:04:50.407819 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:04:50.407790    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790fb00a_b1d5_4e14_8186_8e85874a0fdf.slice/crio-0ec9ebe50f3e35cd15c0c21adabcc18b02583d7274110f3ee75d96c1cc165362 WatchSource:0}: Error finding container 0ec9ebe50f3e35cd15c0c21adabcc18b02583d7274110f3ee75d96c1cc165362: Status 404 returned error can't find the container with id 0ec9ebe50f3e35cd15c0c21adabcc18b02583d7274110f3ee75d96c1cc165362
Jul 13 19:04:50.409630 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.409614    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 19:04:50.532656 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.532565    2569 generic.go:356] "Generic (PLEG): container finished" podID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerID="df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e" exitCode=2
Jul 13 19:04:50.532656 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.532616    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" event={"ID":"e07a79b0-93de-4baf-af00-5cfb9a30a665","Type":"ContainerDied","Data":"df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e"}
Jul 13 19:04:50.534704 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.534679    2569 generic.go:356] "Generic (PLEG): container finished" podID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerID="50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a" exitCode=2
Jul 13 19:04:50.534845 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.534749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" event={"ID":"9fc098c6-da5b-4d00-a45d-9e083e7d8406","Type":"ContainerDied","Data":"50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a"}
Jul 13 19:04:50.535751 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:50.535734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649" event={"ID":"790fb00a-b1d5-4e14-8186-8e85874a0fdf","Type":"ContainerStarted","Data":"0ec9ebe50f3e35cd15c0c21adabcc18b02583d7274110f3ee75d96c1cc165362"}
Jul 13 19:04:52.135960 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:52.135923    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 13 19:04:52.545099 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:52.545000    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649" event={"ID":"790fb00a-b1d5-4e14-8186-8e85874a0fdf","Type":"ContainerStarted","Data":"a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e"}
Jul 13 19:04:52.545272 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:52.545102    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:52.545272 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:52.545137    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649" event={"ID":"790fb00a-b1d5-4e14-8186-8e85874a0fdf","Type":"ContainerStarted","Data":"2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd"}
Jul 13 19:04:52.545272 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:52.545167    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:52.546616 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:52.546596    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:52.567919 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:52.567868    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649" podStartSLOduration=2.430303419 podStartE2EDuration="3.567853695s" totalImagesPullingTime="1.137550276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:04:52.566581389 +0000 UTC m=+884.672185005" watchObservedRunningTime="2026-07-13 19:04:52.567853695 +0000 UTC m=+884.673457314"
Jul 13 19:04:53.137839 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.137793    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 13 19:04:53.142170 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.142140    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 13 19:04:53.896036 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.896007    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:04:53.951463 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.951374    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-cabundle-cert\") pod \"e07a79b0-93de-4baf-af00-5cfb9a30a665\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") "
Jul 13 19:04:53.951463 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.951427    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-rbac-proxy-sar-config\") pod \"e07a79b0-93de-4baf-af00-5cfb9a30a665\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") "
Jul 13 19:04:53.951724 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.951479    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwvmf\" (UniqueName: \"kubernetes.io/projected/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-api-access-bwvmf\") pod \"e07a79b0-93de-4baf-af00-5cfb9a30a665\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") "
Jul 13 19:04:53.951724 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.951572    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e07a79b0-93de-4baf-af00-5cfb9a30a665-kserve-provision-location\") pod \"e07a79b0-93de-4baf-af00-5cfb9a30a665\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") "
Jul 13 19:04:53.951724 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.951622    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e07a79b0-93de-4baf-af00-5cfb9a30a665-proxy-tls\") pod \"e07a79b0-93de-4baf-af00-5cfb9a30a665\" (UID: \"e07a79b0-93de-4baf-af00-5cfb9a30a665\") "
Jul 13 19:04:53.951879 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.951854    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e07a79b0-93de-4baf-af00-5cfb9a30a665" (UID: "e07a79b0-93de-4baf-af00-5cfb9a30a665"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:04:53.951919 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.951849    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e07a79b0-93de-4baf-af00-5cfb9a30a665" (UID: "e07a79b0-93de-4baf-af00-5cfb9a30a665"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:04:53.951955 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.951921    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07a79b0-93de-4baf-af00-5cfb9a30a665-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e07a79b0-93de-4baf-af00-5cfb9a30a665" (UID: "e07a79b0-93de-4baf-af00-5cfb9a30a665"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:04:53.953663 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.953631    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07a79b0-93de-4baf-af00-5cfb9a30a665-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e07a79b0-93de-4baf-af00-5cfb9a30a665" (UID: "e07a79b0-93de-4baf-af00-5cfb9a30a665"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:04:53.953894 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:53.953876    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-api-access-bwvmf" (OuterVolumeSpecName: "kube-api-access-bwvmf") pod "e07a79b0-93de-4baf-af00-5cfb9a30a665" (UID: "e07a79b0-93de-4baf-af00-5cfb9a30a665"). InnerVolumeSpecName "kube-api-access-bwvmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:04:54.052372 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.052350    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e07a79b0-93de-4baf-af00-5cfb9a30a665-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:54.052468 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.052376    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e07a79b0-93de-4baf-af00-5cfb9a30a665-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:54.052468 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.052387    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:54.052468 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.052396    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:54.052468 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.052403    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwvmf\" (UniqueName: \"kubernetes.io/projected/e07a79b0-93de-4baf-af00-5cfb9a30a665-kube-api-access-bwvmf\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:54.555857 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.555820    2569 generic.go:356] "Generic (PLEG): container finished" podID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerID="87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d" exitCode=0
Jul 13 19:04:54.556315 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.555869    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"
Jul 13 19:04:54.556315 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.555910    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" event={"ID":"e07a79b0-93de-4baf-af00-5cfb9a30a665","Type":"ContainerDied","Data":"87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d"}
Jul 13 19:04:54.556315 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.555977    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb" event={"ID":"e07a79b0-93de-4baf-af00-5cfb9a30a665","Type":"ContainerDied","Data":"00215bedc6eb6514f3c51ed2ae6efe680c63177ac0c2fe876d9861ce02af2c0b"}
Jul 13 19:04:54.556315 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.556004    2569 scope.go:117] "RemoveContainer" containerID="df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e"
Jul 13 19:04:54.564022 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.564005    2569 scope.go:117] "RemoveContainer" containerID="87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d"
Jul 13 19:04:54.571569 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.571550    2569 scope.go:117] "RemoveContainer" containerID="8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7"
Jul 13 19:04:54.577404 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.577380    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"]
Jul 13 19:04:54.578903 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.578886    2569 scope.go:117] "RemoveContainer" containerID="df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e"
Jul 13 19:04:54.579163 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:04:54.579147    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e\": container with ID starting with df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e not found: ID does not exist" containerID="df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e"
Jul 13 19:04:54.579217 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.579170    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e"} err="failed to get container status \"df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e\": rpc error: code = NotFound desc = could not find container \"df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e\": container with ID starting with df872d74215520abbbdcc877767dcd247c8df280c0a99a6e6502dabe36e03c1e not found: ID does not exist"
Jul 13 19:04:54.579217 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.579189    2569 scope.go:117] "RemoveContainer" containerID="87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d"
Jul 13 19:04:54.579598 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:04:54.579521    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d\": container with ID starting with 87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d not found: ID does not exist" containerID="87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d"
Jul 13 19:04:54.579715 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.579605    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d"} err="failed to get container status \"87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d\": rpc error: code = NotFound desc = could not find container \"87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d\": container with ID starting with 87c2dac25e7ec8eafe72101be43b3242efcaa194868c6c5064fe30aaeb537e6d not found: ID does not exist"
Jul 13 19:04:54.579715 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.579629    2569 scope.go:117] "RemoveContainer" containerID="8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7"
Jul 13 19:04:54.579948 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:04:54.579926    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7\": container with ID starting with 8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7 not found: ID does not exist" containerID="8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7"
Jul 13 19:04:54.580034 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.579958    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7"} err="failed to get container status \"8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7\": rpc error: code = NotFound desc = could not find container \"8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7\": container with ID starting with 8112b5ec190156e064053f8d1c17e25da3fcb5daae9549ccc77af6e17abd9be7 not found: ID does not exist"
Jul 13 19:04:54.581480 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.581459    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-be515-predictor-85858c546f-jxlvb"]
Jul 13 19:04:54.863919 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.863898    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:04:54.958362 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958333    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6wtf\" (UniqueName: \"kubernetes.io/projected/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-api-access-b6wtf\") pod \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") "
Jul 13 19:04:54.958604 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958385    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-rbac-proxy-sar-config\") pod \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") "
Jul 13 19:04:54.958604 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958501    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kserve-provision-location\") pod \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") "
Jul 13 19:04:54.958604 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958559    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fc098c6-da5b-4d00-a45d-9e083e7d8406-proxy-tls\") pod \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") "
Jul 13 19:04:54.958740 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958615    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-cabundle-cert\") pod \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\" (UID: \"9fc098c6-da5b-4d00-a45d-9e083e7d8406\") "
Jul 13 19:04:54.958807 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958771    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fc098c6-da5b-4d00-a45d-9e083e7d8406" (UID: "9fc098c6-da5b-4d00-a45d-9e083e7d8406"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:04:54.958807 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958773    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9fc098c6-da5b-4d00-a45d-9e083e7d8406" (UID: "9fc098c6-da5b-4d00-a45d-9e083e7d8406"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:04:54.958921 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958850    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:54.958921 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958868    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:54.959027 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.958953    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9fc098c6-da5b-4d00-a45d-9e083e7d8406" (UID: "9fc098c6-da5b-4d00-a45d-9e083e7d8406"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:04:54.960500 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.960481    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-api-access-b6wtf" (OuterVolumeSpecName: "kube-api-access-b6wtf") pod "9fc098c6-da5b-4d00-a45d-9e083e7d8406" (UID: "9fc098c6-da5b-4d00-a45d-9e083e7d8406"). InnerVolumeSpecName "kube-api-access-b6wtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:04:54.960624 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:54.960544    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc098c6-da5b-4d00-a45d-9e083e7d8406-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fc098c6-da5b-4d00-a45d-9e083e7d8406" (UID: "9fc098c6-da5b-4d00-a45d-9e083e7d8406"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:04:55.059744 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.059710    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fc098c6-da5b-4d00-a45d-9e083e7d8406-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:55.059744 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.059739    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc098c6-da5b-4d00-a45d-9e083e7d8406-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:55.059744 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.059750    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6wtf\" (UniqueName: \"kubernetes.io/projected/9fc098c6-da5b-4d00-a45d-9e083e7d8406-kube-api-access-b6wtf\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:04:55.562169 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.562133    2569 generic.go:356] "Generic (PLEG): container finished" podID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerID="f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4" exitCode=0
Jul 13 19:04:55.562631 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.562187    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"
Jul 13 19:04:55.562631 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.562218    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" event={"ID":"9fc098c6-da5b-4d00-a45d-9e083e7d8406","Type":"ContainerDied","Data":"f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4"}
Jul 13 19:04:55.562631 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.562283    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw" event={"ID":"9fc098c6-da5b-4d00-a45d-9e083e7d8406","Type":"ContainerDied","Data":"f6d59c699af7b5894bd40b748b57808b8d77e37cb926f2b9e6223bbf53156eb0"}
Jul 13 19:04:55.562631 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.562314    2569 scope.go:117] "RemoveContainer" containerID="50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a"
Jul 13 19:04:55.570292 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.570276    2569 scope.go:117] "RemoveContainer" containerID="f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4"
Jul 13 19:04:55.577563 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.577519    2569 scope.go:117] "RemoveContainer" containerID="6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625"
Jul 13 19:04:55.584162 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.584146    2569 scope.go:117] "RemoveContainer" containerID="50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a"
Jul 13 19:04:55.584386 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:04:55.584366    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a\": container with ID starting with 50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a not found: ID does not exist" containerID="50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a"
Jul 13 19:04:55.584509 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.584390    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a"} err="failed to get container status \"50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a\": rpc error: code = NotFound desc = could not find container \"50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a\": container with ID starting with 50e1f81fcf38b4dd58b79a124e63c374a67d612ac2ec506f4a77524cb45fd62a not found: ID does not exist"
Jul 13 19:04:55.584509 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.584407    2569 scope.go:117] "RemoveContainer" containerID="f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4"
Jul 13 19:04:55.584776 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:04:55.584755    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4\": container with ID starting with f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4 not found: ID does not exist" containerID="f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4"
Jul 13 19:04:55.584846 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.584790    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4"} err="failed to get container status \"f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4\": rpc error: code = NotFound desc = could not find container \"f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4\": container with ID starting with f73630d8226dc662260cfc74b26163778efd2abbc166bcf4c3b32a4e196bebe4 not found: ID does not exist"
Jul 13 19:04:55.584846 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.584804    2569 scope.go:117] "RemoveContainer" containerID="6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625"
Jul 13 19:04:55.585131 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:04:55.585103    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625\": container with ID starting with 6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625 not found: ID does not exist" containerID="6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625"
Jul 13 19:04:55.585214 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.585135    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625"} err="failed to get container status \"6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625\": rpc error: code = NotFound desc = could not find container \"6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625\": container with ID starting with 6762317b158c7b2db9187558b9e309ff4f2436240dcc13a501dca92edfa00625 not found: ID does not exist"
Jul 13 19:04:55.587283 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.587262    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"]
Jul 13 19:04:55.594198 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:55.594177    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-be515-predictor-7d6cc8854c-wpvbw"]
Jul 13 19:04:56.045206 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:56.045169    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" path="/var/lib/kubelet/pods/9fc098c6-da5b-4d00-a45d-9e083e7d8406/volumes"
Jul 13 19:04:56.045694 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:56.045680    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" path="/var/lib/kubelet/pods/e07a79b0-93de-4baf-af00-5cfb9a30a665/volumes"
Jul 13 19:04:59.561017 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.560987    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:04:59.934779 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.934745    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:04:59.981428 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981392    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"]
Jul 13 19:04:59.981766 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981749    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="storage-initializer"
Jul 13 19:04:59.981853 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981769    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="storage-initializer"
Jul 13 19:04:59.981853 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981784    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="storage-initializer"
Jul 13 19:04:59.981853 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981792    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="storage-initializer"
Jul 13 19:04:59.981853 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981805    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container"
Jul 13 19:04:59.981853 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981813    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container"
Jul 13 19:04:59.981853 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981823    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kube-rbac-proxy"
Jul 13 19:04:59.981853 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981831    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kube-rbac-proxy"
Jul 13 19:04:59.981853 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981841    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container"
Jul 13 19:04:59.981853 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981849    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container"
Jul 13 19:04:59.982298 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981878    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kube-rbac-proxy"
Jul 13 19:04:59.982298 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981889    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kube-rbac-proxy"
Jul 13 19:04:59.982298 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981964    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kserve-container"
Jul 13 19:04:59.982298 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981977    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc098c6-da5b-4d00-a45d-9e083e7d8406" containerName="kube-rbac-proxy"
Jul 13 19:04:59.982298 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981989    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kube-rbac-proxy"
Jul 13 19:04:59.982298 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.981999    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07a79b0-93de-4baf-af00-5cfb9a30a665" containerName="kserve-container"
Jul 13 19:04:59.986800 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.986779    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:04:59.990462 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.990443    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-2a04c-predictor-serving-cert\""
Jul 13 19:04:59.990642 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.990626    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:04:59.991642 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.991623    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 19:04:59.991776 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.991761    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-2a04c-kube-rbac-proxy-sar-config\""
Jul 13 19:04:59.999020 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:04:59.999000    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"]
Jul 13 19:05:00.101590 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.101504    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01dc1117-967a-4c02-a616-a014111171ea-proxy-tls\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.101590 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.101578    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc7jz\" (UniqueName: \"kubernetes.io/projected/01dc1117-967a-4c02-a616-a014111171ea-kube-api-access-mc7jz\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.101774 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.101645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.101774 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.101685    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01dc1117-967a-4c02-a616-a014111171ea-kserve-provision-location\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.101774 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.101721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-cabundle-cert\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.194632 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.194514    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:00.202765 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.202745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01dc1117-967a-4c02-a616-a014111171ea-proxy-tls\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.202847 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.202786    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc7jz\" (UniqueName: \"kubernetes.io/projected/01dc1117-967a-4c02-a616-a014111171ea-kube-api-access-mc7jz\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.202919 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.202901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.202961 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.202940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01dc1117-967a-4c02-a616-a014111171ea-kserve-provision-location\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.203014 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.202967    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-cabundle-cert\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.203357 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.203336    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01dc1117-967a-4c02-a616-a014111171ea-kserve-provision-location\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.203620 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.203603    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.203668 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.203633    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-cabundle-cert\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.205161 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.205145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01dc1117-967a-4c02-a616-a014111171ea-proxy-tls\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.215202 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.215177    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc7jz\" (UniqueName: \"kubernetes.io/projected/01dc1117-967a-4c02-a616-a014111171ea-kube-api-access-mc7jz\") pod \"isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.297220 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.297176    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:00.452589 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.452563    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"]
Jul 13 19:05:00.455170 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:05:00.455137    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dc1117_967a_4c02_a616_a014111171ea.slice/crio-91f1ac5e24ea9653039ab081896479a38ae113e9de3efd0149c3732d74227f4e WatchSource:0}: Error finding container 91f1ac5e24ea9653039ab081896479a38ae113e9de3efd0149c3732d74227f4e: Status 404 returned error can't find the container with id 91f1ac5e24ea9653039ab081896479a38ae113e9de3efd0149c3732d74227f4e
Jul 13 19:05:00.585606 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.585572    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerStarted","Data":"f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537"}
Jul 13 19:05:00.585943 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:00.585639    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerStarted","Data":"91f1ac5e24ea9653039ab081896479a38ae113e9de3efd0149c3732d74227f4e"}
Jul 13 19:05:04.604148 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:04.604065    2569 generic.go:356] "Generic (PLEG): container finished" podID="01dc1117-967a-4c02-a616-a014111171ea" containerID="f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537" exitCode=0
Jul 13 19:05:04.604148 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:04.604138    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerDied","Data":"f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537"}
Jul 13 19:05:05.610698 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:05.610663    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerStarted","Data":"dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b"}
Jul 13 19:05:05.611187 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:05.610732    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerStarted","Data":"ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b"}
Jul 13 19:05:05.611187 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:05.610756    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerStarted","Data":"b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5"}
Jul 13 19:05:05.611187 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:05.611086    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:05.611187 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:05.611158    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:05.612338 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:05.612311    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:05:05.634066 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:05.634012    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podStartSLOduration=2.7919211 podStartE2EDuration="6.633995043s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.842073943s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:05.632636342 +0000 UTC m=+897.738239962" watchObservedRunningTime="2026-07-13 19:05:05.633995043 +0000 UTC m=+897.739598665"
Jul 13 19:05:06.614711 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:06.614678    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:06.615179 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:06.614678    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:05:06.615802 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:06.615774    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:05:07.618869 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:07.618824    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:05:07.619242 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:07.619204    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:05:09.923844 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:09.923813    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 19:05:09.924300 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:09.924263    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 19:05:12.622936 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:12.622906    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:05:12.625323 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:12.623398    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:05:12.625323 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:12.623881    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:05:22.624040 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:22.623992    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:05:22.624631 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:22.624501    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:05:32.623790 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:32.623741    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:05:32.624259 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:32.624182    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:05:42.623836 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:42.623794    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:05:42.624346 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:42.624323    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:05:45.196939 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:45.196904    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:47.214659 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:47.214610    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:05:52.623488 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:52.623445    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:05:52.623984 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:05:52.623919    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:06:02.623966 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:02.623926    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:06:02.624550 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:02.624504    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:06:12.623432 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:12.623379    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:06:12.623897 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:12.623869    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:06:22.624258 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:22.624227    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:06:22.624714 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:22.624515    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:06:28.362130 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:28.362099    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:34.973440 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:34.973408    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-2a04c-predictor-6bfbf69478-hm649_790fb00a-b1d5-4e14-8186-8e85874a0fdf/kserve-container/0.log"
Jul 13 19:06:35.185699 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.185660    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"]
Jul 13 19:06:35.186128 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.186096    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" containerID="cri-o://b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5" gracePeriod=30
Jul 13 19:06:35.188286 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.186345    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" containerID="cri-o://dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b" gracePeriod=30
Jul 13 19:06:35.188286 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.186384    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy" containerID="cri-o://ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b" gracePeriod=30
Jul 13 19:06:35.219418 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.219397    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:35.280639 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.280553    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"]
Jul 13 19:06:35.280874 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.280827    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649" podUID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerName="kserve-container" containerID="cri-o://2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd" gracePeriod=30
Jul 13 19:06:35.280936 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.280908    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649" podUID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerName="kube-rbac-proxy" containerID="cri-o://a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e" gracePeriod=30
Jul 13 19:06:35.304634 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.304607    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:35.363690 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.363656    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"]
Jul 13 19:06:35.366869 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.366834    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.369429 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.369410    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b1b9c-predictor-serving-cert\""
Jul 13 19:06:35.369677 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.369663    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b1b9c-kube-rbac-proxy-sar-config\""
Jul 13 19:06:35.383826 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.383804    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"]
Jul 13 19:06:35.387819 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.387722    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:35.465608 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.465572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41206af-1ae6-4ba5-bf1e-14f15b509f23-proxy-tls\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.465780 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.465639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz86v\" (UniqueName: \"kubernetes.io/projected/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-api-access-gz86v\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.465780 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.465672    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.465780 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.465712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.465780 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.465742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.529851 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.529825    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:06:35.566726 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.566650    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790fb00a-b1d5-4e14-8186-8e85874a0fdf-proxy-tls\") pod \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") "
Jul 13 19:06:35.566726 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.566720    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q9h5\" (UniqueName: \"kubernetes.io/projected/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-api-access-5q9h5\") pod \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") "
Jul 13 19:06:35.566941 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.566754    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-rbac-proxy-sar-config\") pod \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\" (UID: \"790fb00a-b1d5-4e14-8186-8e85874a0fdf\") "
Jul 13 19:06:35.567001 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.566937    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41206af-1ae6-4ba5-bf1e-14f15b509f23-proxy-tls\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.567001 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.566989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz86v\" (UniqueName: \"kubernetes.io/projected/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-api-access-gz86v\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.567101 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.567024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.567101 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.567066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.567273 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:06:35.567094    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-serving-cert: secret "isvc-sklearn-scale-raw-b1b9c-predictor-serving-cert" not found
Jul 13 19:06:35.567273 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.567145    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "790fb00a-b1d5-4e14-8186-8e85874a0fdf" (UID: "790fb00a-b1d5-4e14-8186-8e85874a0fdf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:06:35.567273 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:06:35.567155    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41206af-1ae6-4ba5-bf1e-14f15b509f23-proxy-tls podName:d41206af-1ae6-4ba5-bf1e-14f15b509f23 nodeName:}" failed. No retries permitted until 2026-07-13 19:06:36.067137155 +0000 UTC m=+988.172740752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d41206af-1ae6-4ba5-bf1e-14f15b509f23-proxy-tls") pod "isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" (UID: "d41206af-1ae6-4ba5-bf1e-14f15b509f23") : secret "isvc-sklearn-scale-raw-b1b9c-predictor-serving-cert" not found
Jul 13 19:06:35.567273 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.567098    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.567273 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.567254    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:06:35.567636 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.567600    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.567849 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.567829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.567999 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.567972    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.568927 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.568902    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790fb00a-b1d5-4e14-8186-8e85874a0fdf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "790fb00a-b1d5-4e14-8186-8e85874a0fdf" (UID: "790fb00a-b1d5-4e14-8186-8e85874a0fdf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:06:35.569297 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.569271    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-api-access-5q9h5" (OuterVolumeSpecName: "kube-api-access-5q9h5") pod "790fb00a-b1d5-4e14-8186-8e85874a0fdf" (UID: "790fb00a-b1d5-4e14-8186-8e85874a0fdf"). InnerVolumeSpecName "kube-api-access-5q9h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:06:35.577738 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.577718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz86v\" (UniqueName: \"kubernetes.io/projected/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-api-access-gz86v\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:35.668142 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.668114    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790fb00a-b1d5-4e14-8186-8e85874a0fdf-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:06:35.668142 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.668140    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q9h5\" (UniqueName: \"kubernetes.io/projected/790fb00a-b1d5-4e14-8186-8e85874a0fdf-kube-api-access-5q9h5\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:06:35.669107 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.669085    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:35.948315 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.948284    2569 generic.go:356] "Generic (PLEG): container finished" podID="01dc1117-967a-4c02-a616-a014111171ea" containerID="ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b" exitCode=2
Jul 13 19:06:35.948493 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.948353    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerDied","Data":"ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b"}
Jul 13 19:06:35.949849 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.949825    2569 generic.go:356] "Generic (PLEG): container finished" podID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerID="a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e" exitCode=2
Jul 13 19:06:35.949849 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.949848    2569 generic.go:356] "Generic (PLEG): container finished" podID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerID="2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd" exitCode=2
Jul 13 19:06:35.950018 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.949869    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649" event={"ID":"790fb00a-b1d5-4e14-8186-8e85874a0fdf","Type":"ContainerDied","Data":"a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e"}
Jul 13 19:06:35.950018 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.949881    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"
Jul 13 19:06:35.950018 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.949922    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649" event={"ID":"790fb00a-b1d5-4e14-8186-8e85874a0fdf","Type":"ContainerDied","Data":"2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd"}
Jul 13 19:06:35.950018 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.949953    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649" event={"ID":"790fb00a-b1d5-4e14-8186-8e85874a0fdf","Type":"ContainerDied","Data":"0ec9ebe50f3e35cd15c0c21adabcc18b02583d7274110f3ee75d96c1cc165362"}
Jul 13 19:06:35.950018 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.949989    2569 scope.go:117] "RemoveContainer" containerID="a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e"
Jul 13 19:06:35.958888 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.958870    2569 scope.go:117] "RemoveContainer" containerID="2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd"
Jul 13 19:06:35.965903 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.965889    2569 scope.go:117] "RemoveContainer" containerID="a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e"
Jul 13 19:06:35.966128 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:06:35.966111    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e\": container with ID starting with a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e not found: ID does not exist" containerID="a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e"
Jul 13 19:06:35.966176 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.966136    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e"} err="failed to get container status \"a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e\": rpc error: code = NotFound desc = could not find container \"a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e\": container with ID starting with a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e not found: ID does not exist"
Jul 13 19:06:35.966176 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.966151    2569 scope.go:117] "RemoveContainer" containerID="2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd"
Jul 13 19:06:35.966356 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:06:35.966340    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd\": container with ID starting with 2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd not found: ID does not exist" containerID="2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd"
Jul 13 19:06:35.966398 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.966363    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd"} err="failed to get container status \"2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd\": rpc error: code = NotFound desc = could not find container \"2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd\": container with ID starting with 2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd not found: ID does not exist"
Jul 13 19:06:35.966398 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.966380    2569 scope.go:117] "RemoveContainer" containerID="a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e"
Jul 13 19:06:35.966631 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.966614    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e"} err="failed to get container status \"a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e\": rpc error: code = NotFound desc = could not find container \"a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e\": container with ID starting with a88d8e97e60b03a62627165f4851920ff4f0e4ee244866ab94e9847f74e87d0e not found: ID does not exist"
Jul 13 19:06:35.966631 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.966631    2569 scope.go:117] "RemoveContainer" containerID="2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd"
Jul 13 19:06:35.966869 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.966851    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd"} err="failed to get container status \"2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd\": rpc error: code = NotFound desc = could not find container \"2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd\": container with ID starting with 2bf5abfc0ffa9c6af38dd28b8d61a6dd22d4e48ab934bc702284dfe622a838cd not found: ID does not exist"
Jul 13 19:06:35.973398 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.973379    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"]
Jul 13 19:06:35.978454 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:35.978430    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2a04c-predictor-6bfbf69478-hm649"]
Jul 13 19:06:36.045567 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:36.045518    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" path="/var/lib/kubelet/pods/790fb00a-b1d5-4e14-8186-8e85874a0fdf/volumes"
Jul 13 19:06:36.072017 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:36.071994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41206af-1ae6-4ba5-bf1e-14f15b509f23-proxy-tls\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:36.074215 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:36.074196    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41206af-1ae6-4ba5-bf1e-14f15b509f23-proxy-tls\") pod \"isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:36.276999 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:36.276959    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:36.424822 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:36.424799    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"]
Jul 13 19:06:36.427119 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:06:36.427089    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41206af_1ae6_4ba5_bf1e_14f15b509f23.slice/crio-c4bb1a71f6c0b07166ae7dc3cbdb43d9ff67e2af5520e98079ce124fa822065a WatchSource:0}: Error finding container c4bb1a71f6c0b07166ae7dc3cbdb43d9ff67e2af5520e98079ce124fa822065a: Status 404 returned error can't find the container with id c4bb1a71f6c0b07166ae7dc3cbdb43d9ff67e2af5520e98079ce124fa822065a
Jul 13 19:06:36.955407 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:36.955365    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" event={"ID":"d41206af-1ae6-4ba5-bf1e-14f15b509f23","Type":"ContainerStarted","Data":"270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291"}
Jul 13 19:06:36.955634 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:36.955436    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" event={"ID":"d41206af-1ae6-4ba5-bf1e-14f15b509f23","Type":"ContainerStarted","Data":"c4bb1a71f6c0b07166ae7dc3cbdb43d9ff67e2af5520e98079ce124fa822065a"}
Jul 13 19:06:37.619847 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:37.619809    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 13 19:06:39.970111 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:39.970080    2569 generic.go:356] "Generic (PLEG): container finished" podID="01dc1117-967a-4c02-a616-a014111171ea" containerID="b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5" exitCode=0
Jul 13 19:06:39.970438 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:39.970125    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerDied","Data":"b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5"}
Jul 13 19:06:40.975019 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:40.974983    2569 generic.go:356] "Generic (PLEG): container finished" podID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerID="270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291" exitCode=0
Jul 13 19:06:40.975395 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:40.975031    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" event={"ID":"d41206af-1ae6-4ba5-bf1e-14f15b509f23","Type":"ContainerDied","Data":"270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291"}
Jul 13 19:06:41.981103 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:41.981066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" event={"ID":"d41206af-1ae6-4ba5-bf1e-14f15b509f23","Type":"ContainerStarted","Data":"46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a"}
Jul 13 19:06:41.981524 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:41.981127    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" event={"ID":"d41206af-1ae6-4ba5-bf1e-14f15b509f23","Type":"ContainerStarted","Data":"ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c"}
Jul 13 19:06:41.981524 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:41.981415    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:41.981524 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:41.981477    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:41.982835 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:41.982807    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:06:42.035424 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:42.035373    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podStartSLOduration=3.345139582 podStartE2EDuration="7.035358086s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690218504s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:06:42.030263053 +0000 UTC m=+994.135866671" watchObservedRunningTime="2026-07-13 19:06:42.035358086 +0000 UTC m=+994.140961705"
Jul 13 19:06:42.619742 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:42.619701    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 13 19:06:42.624091 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:42.624063    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:06:42.624468 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:42.624443    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:06:42.984664 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:42.984626    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:06:47.619192 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:47.619155    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 13 19:06:47.619616 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:47.619287    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:06:47.988702 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:47.988672    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:06:47.989198 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:47.989171    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:06:51.529339 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:51.529307    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:06:52.619828 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:52.619783    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 13 19:06:52.624159 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:52.624133    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:06:52.624544 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:52.624499    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:06:57.619202 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:57.619158    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 13 19:06:57.989113 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:06:57.989077    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:07:02.619834 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:02.619781    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 13 19:07:02.624246 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:02.624210    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 13 19:07:02.624389 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:02.624343    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:07:02.624634 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:02.624608    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 19:07:02.624724 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:02.624713    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:07:05.338934 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.338911    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:07:05.415788 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.415754    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc7jz\" (UniqueName: \"kubernetes.io/projected/01dc1117-967a-4c02-a616-a014111171ea-kube-api-access-mc7jz\") pod \"01dc1117-967a-4c02-a616-a014111171ea\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") "
Jul 13 19:07:05.415953 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.415798    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01dc1117-967a-4c02-a616-a014111171ea-proxy-tls\") pod \"01dc1117-967a-4c02-a616-a014111171ea\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") "
Jul 13 19:07:05.415953 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.415853    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-cabundle-cert\") pod \"01dc1117-967a-4c02-a616-a014111171ea\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") "
Jul 13 19:07:05.415953 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.415898    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-kube-rbac-proxy-sar-config\") pod \"01dc1117-967a-4c02-a616-a014111171ea\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") "
Jul 13 19:07:05.415953 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.415938    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01dc1117-967a-4c02-a616-a014111171ea-kserve-provision-location\") pod \"01dc1117-967a-4c02-a616-a014111171ea\" (UID: \"01dc1117-967a-4c02-a616-a014111171ea\") "
Jul 13 19:07:05.416256 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.416228    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01dc1117-967a-4c02-a616-a014111171ea" (UID: "01dc1117-967a-4c02-a616-a014111171ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:07:05.416443 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.416268    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01dc1117-967a-4c02-a616-a014111171ea" (UID: "01dc1117-967a-4c02-a616-a014111171ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:07:05.416443 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.416306    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dc1117-967a-4c02-a616-a014111171ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01dc1117-967a-4c02-a616-a014111171ea" (UID: "01dc1117-967a-4c02-a616-a014111171ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:07:05.417949 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.417927    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dc1117-967a-4c02-a616-a014111171ea-kube-api-access-mc7jz" (OuterVolumeSpecName: "kube-api-access-mc7jz") pod "01dc1117-967a-4c02-a616-a014111171ea" (UID: "01dc1117-967a-4c02-a616-a014111171ea"). InnerVolumeSpecName "kube-api-access-mc7jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:07:05.418018 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.417976    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01dc1117-967a-4c02-a616-a014111171ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01dc1117-967a-4c02-a616-a014111171ea" (UID: "01dc1117-967a-4c02-a616-a014111171ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:07:05.517382 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.517310    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc7jz\" (UniqueName: \"kubernetes.io/projected/01dc1117-967a-4c02-a616-a014111171ea-kube-api-access-mc7jz\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:07:05.517382 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.517337    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01dc1117-967a-4c02-a616-a014111171ea-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:07:05.517382 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.517348    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:07:05.517382 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.517356    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01dc1117-967a-4c02-a616-a014111171ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:07:05.517382 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:05.517364    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01dc1117-967a-4c02-a616-a014111171ea-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:07:06.074418 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.074383    2569 generic.go:356] "Generic (PLEG): container finished" podID="01dc1117-967a-4c02-a616-a014111171ea" containerID="dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b" exitCode=0
Jul 13 19:07:06.074574 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.074453    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"
Jul 13 19:07:06.074574 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.074462    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerDied","Data":"dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b"}
Jul 13 19:07:06.074574 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.074547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z" event={"ID":"01dc1117-967a-4c02-a616-a014111171ea","Type":"ContainerDied","Data":"91f1ac5e24ea9653039ab081896479a38ae113e9de3efd0149c3732d74227f4e"}
Jul 13 19:07:06.074696 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.074580    2569 scope.go:117] "RemoveContainer" containerID="dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b"
Jul 13 19:07:06.082330 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.082312    2569 scope.go:117] "RemoveContainer" containerID="ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b"
Jul 13 19:07:06.089404 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.089387    2569 scope.go:117] "RemoveContainer" containerID="b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5"
Jul 13 19:07:06.096605 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.096587    2569 scope.go:117] "RemoveContainer" containerID="f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537"
Jul 13 19:07:06.103349 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.103333    2569 scope.go:117] "RemoveContainer" containerID="dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b"
Jul 13 19:07:06.103623 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:07:06.103604    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b\": container with ID starting with dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b not found: ID does not exist" containerID="dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b"
Jul 13 19:07:06.103688 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.103641    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b"} err="failed to get container status \"dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b\": rpc error: code = NotFound desc = could not find container \"dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b\": container with ID starting with dfcdee264841c2dc00e86bc8e6d81f3c58e37b23d32f1e5e3967310a7b520c3b not found: ID does not exist"
Jul 13 19:07:06.103688 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.103658    2569 scope.go:117] "RemoveContainer" containerID="ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b"
Jul 13 19:07:06.103892 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:07:06.103872    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b\": container with ID starting with ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b not found: ID does not exist" containerID="ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b"
Jul 13 19:07:06.103958 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.103901    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b"} err="failed to get container status \"ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b\": rpc error: code = NotFound desc = could not find container \"ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b\": container with ID starting with ff981e3b21f04b75b5d73f145eb3452db40de4d4edf20449869118e5d0a9446b not found: ID does not exist"
Jul 13 19:07:06.103958 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.103923    2569 scope.go:117] "RemoveContainer" containerID="b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5"
Jul 13 19:07:06.104162 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:07:06.104146    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5\": container with ID starting with b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5 not found: ID does not exist" containerID="b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5"
Jul 13 19:07:06.104200 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.104167    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5"} err="failed to get container status \"b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5\": rpc error: code = NotFound desc = could not find container \"b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5\": container with ID starting with b25b0201c4d51d25a8da9417624bb6d2035d2d093026eb48b186ca6fdd1b77d5 not found: ID does not exist"
Jul 13 19:07:06.104200 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.104179    2569 scope.go:117] "RemoveContainer" containerID="f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537"
Jul 13 19:07:06.104415 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:07:06.104398    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537\": container with ID starting with f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537 not found: ID does not exist" containerID="f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537"
Jul 13 19:07:06.104468 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.104422    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537"} err="failed to get container status \"f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537\": rpc error: code = NotFound desc = could not find container \"f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537\": container with ID starting with f4e5057bc9ac577c84d1e01938fe66f1aa236c286b0816f174b975b5a9a38537 not found: ID does not exist"
Jul 13 19:07:06.117282 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.117259    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"]
Jul 13 19:07:06.123186 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:06.123166    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2a04c-predictor-644c6759b8-89t6z"]
Jul 13 19:07:07.989448 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:07.989409    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:07:08.045446 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:08.045414    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01dc1117-967a-4c02-a616-a014111171ea" path="/var/lib/kubelet/pods/01dc1117-967a-4c02-a616-a014111171ea/volumes"
Jul 13 19:07:17.989318 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:17.989273    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:07:27.989684 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:27.989644    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:07:37.989410 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:37.989368    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:07:47.989416 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:47.989373    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:07:49.041169 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:49.041127    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:07:59.041921 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:07:59.041884    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:08:09.041185 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:08:09.041143    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:08:19.041172 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:08:19.041131    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:08:29.041965 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:08:29.041922    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:08:39.041195 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:08:39.041154    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:08:49.041221 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:08:49.041174    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:08:54.041032 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:08:54.040994    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:09:04.041010 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:04.040969    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:09:14.045575 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:14.045549    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:09:15.478582 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:15.478548    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"]
Jul 13 19:09:15.479004 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:15.478753    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" containerID="cri-o://ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c" gracePeriod=30
Jul 13 19:09:15.479004 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:15.478833    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kube-rbac-proxy" containerID="cri-o://46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a" gracePeriod=30
Jul 13 19:09:15.531282 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:15.531243    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:16.552596 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:16.552566    2569 generic.go:356] "Generic (PLEG): container finished" podID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerID="46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a" exitCode=2
Jul 13 19:09:16.552596 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:16.552601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" event={"ID":"d41206af-1ae6-4ba5-bf1e-14f15b509f23","Type":"ContainerDied","Data":"46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a"}
Jul 13 19:09:17.985092 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:17.985047    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 13 19:09:20.609603 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.609558    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:20.678407 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678368    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"]
Jul 13 19:09:20.678726 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678713    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy"
Jul 13 19:09:20.678781 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678729    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy"
Jul 13 19:09:20.678781 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678745    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerName="kserve-container"
Jul 13 19:09:20.678781 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678751    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerName="kserve-container"
Jul 13 19:09:20.678781 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678761    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent"
Jul 13 19:09:20.678781 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678766    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent"
Jul 13 19:09:20.678781 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678777    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerName="kube-rbac-proxy"
Jul 13 19:09:20.678781 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678783    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerName="kube-rbac-proxy"
Jul 13 19:09:20.679002 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678792    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="storage-initializer"
Jul 13 19:09:20.679002 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678797    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="storage-initializer"
Jul 13 19:09:20.679002 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678806    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container"
Jul 13 19:09:20.679002 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678811    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container"
Jul 13 19:09:20.679002 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678863    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerName="kube-rbac-proxy"
Jul 13 19:09:20.679002 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678871    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="790fb00a-b1d5-4e14-8186-8e85874a0fdf" containerName="kserve-container"
Jul 13 19:09:20.679002 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678879    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kserve-container"
Jul 13 19:09:20.679002 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678885    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="agent"
Jul 13 19:09:20.679002 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.678890    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01dc1117-967a-4c02-a616-a014111171ea" containerName="kube-rbac-proxy"
Jul 13 19:09:20.682094 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.682077    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.685071 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.685048    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-985ff8-predictor-serving-cert\""
Jul 13 19:09:20.685630 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.685609    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-985ff8-kube-rbac-proxy-sar-config\""
Jul 13 19:09:20.697269 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.697242    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"]
Jul 13 19:09:20.805794 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.805750    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/727c813c-bf1b-465a-ae4a-b37a12240845-proxy-tls\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.805978 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.805803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727c813c-bf1b-465a-ae4a-b37a12240845-kserve-provision-location\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.805978 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.805830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s49s5\" (UniqueName: \"kubernetes.io/projected/727c813c-bf1b-465a-ae4a-b37a12240845-kube-api-access-s49s5\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.805978 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.805887    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-kube-rbac-proxy-sar-config\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.805978 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.805905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-cabundle-cert\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.906418 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.906387    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727c813c-bf1b-465a-ae4a-b37a12240845-kserve-provision-location\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.906609 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.906426    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s49s5\" (UniqueName: \"kubernetes.io/projected/727c813c-bf1b-465a-ae4a-b37a12240845-kube-api-access-s49s5\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.906609 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.906572    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-kube-rbac-proxy-sar-config\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.906697 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.906613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-cabundle-cert\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.906697 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.906690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/727c813c-bf1b-465a-ae4a-b37a12240845-proxy-tls\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.906869 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.906848    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727c813c-bf1b-465a-ae4a-b37a12240845-kserve-provision-location\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.907180 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.907161    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-kube-rbac-proxy-sar-config\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.907235 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.907207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-cabundle-cert\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.909099 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.909078    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/727c813c-bf1b-465a-ae4a-b37a12240845-proxy-tls\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.915656 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.915636    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s49s5\" (UniqueName: \"kubernetes.io/projected/727c813c-bf1b-465a-ae4a-b37a12240845-kube-api-access-s49s5\") pod \"isvc-primary-985ff8-predictor-67fb957c74-z66gl\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") " pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:20.916088 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.916068    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:09:20.992045 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:20.992002    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:21.123837 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:21.123811    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"]
Jul 13 19:09:21.125823 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:09:21.125802    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727c813c_bf1b_465a_ae4a_b37a12240845.slice/crio-bd6445acf6a70952c78edbf13c553e33f4d604317a1dbe1bff9df27063cb0d62 WatchSource:0}: Error finding container bd6445acf6a70952c78edbf13c553e33f4d604317a1dbe1bff9df27063cb0d62: Status 404 returned error can't find the container with id bd6445acf6a70952c78edbf13c553e33f4d604317a1dbe1bff9df27063cb0d62
Jul 13 19:09:21.572189 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:21.572086    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" event={"ID":"727c813c-bf1b-465a-ae4a-b37a12240845","Type":"ContainerStarted","Data":"c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22"}
Jul 13 19:09:21.572189 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:21.572146    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" event={"ID":"727c813c-bf1b-465a-ae4a-b37a12240845","Type":"ContainerStarted","Data":"bd6445acf6a70952c78edbf13c553e33f4d604317a1dbe1bff9df27063cb0d62"}
Jul 13 19:09:22.985817 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:22.985773    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 13 19:09:24.041882 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:24.041841    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 13 19:09:25.227026 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.226966    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:09:25.346505 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.346472    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-cabundle-cert\") pod \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") "
Jul 13 19:09:25.346694 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.346514    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz86v\" (UniqueName: \"kubernetes.io/projected/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-api-access-gz86v\") pod \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") "
Jul 13 19:09:25.346694 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.346548    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-rbac-proxy-sar-config\") pod \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") "
Jul 13 19:09:25.346694 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.346603    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41206af-1ae6-4ba5-bf1e-14f15b509f23-proxy-tls\") pod \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") "
Jul 13 19:09:25.346694 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.346650    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kserve-provision-location\") pod \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\" (UID: \"d41206af-1ae6-4ba5-bf1e-14f15b509f23\") "
Jul 13 19:09:25.346929 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.346890    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d41206af-1ae6-4ba5-bf1e-14f15b509f23" (UID: "d41206af-1ae6-4ba5-bf1e-14f15b509f23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:09:25.347023 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.346998    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d41206af-1ae6-4ba5-bf1e-14f15b509f23" (UID: "d41206af-1ae6-4ba5-bf1e-14f15b509f23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:09:25.347023 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.347010    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d41206af-1ae6-4ba5-bf1e-14f15b509f23" (UID: "d41206af-1ae6-4ba5-bf1e-14f15b509f23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:09:25.348712 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.348689    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-api-access-gz86v" (OuterVolumeSpecName: "kube-api-access-gz86v") pod "d41206af-1ae6-4ba5-bf1e-14f15b509f23" (UID: "d41206af-1ae6-4ba5-bf1e-14f15b509f23"). InnerVolumeSpecName "kube-api-access-gz86v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:09:25.348793 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.348690    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d41206af-1ae6-4ba5-bf1e-14f15b509f23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d41206af-1ae6-4ba5-bf1e-14f15b509f23" (UID: "d41206af-1ae6-4ba5-bf1e-14f15b509f23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:09:25.447212 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.447173    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41206af-1ae6-4ba5-bf1e-14f15b509f23-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:09:25.447212 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.447203    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:09:25.447212 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.447212    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:09:25.447212 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.447222    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz86v\" (UniqueName: \"kubernetes.io/projected/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-api-access-gz86v\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:09:25.447471 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.447231    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41206af-1ae6-4ba5-bf1e-14f15b509f23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:09:25.589251 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.589174    2569 generic.go:356] "Generic (PLEG): container finished" podID="727c813c-bf1b-465a-ae4a-b37a12240845" containerID="c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22" exitCode=0
Jul 13 19:09:25.589391 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.589243    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" event={"ID":"727c813c-bf1b-465a-ae4a-b37a12240845","Type":"ContainerDied","Data":"c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22"}
Jul 13 19:09:25.591192 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.591168    2569 generic.go:356] "Generic (PLEG): container finished" podID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerID="ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c" exitCode=0
Jul 13 19:09:25.591295 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.591215    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"
Jul 13 19:09:25.591295 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.591234    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" event={"ID":"d41206af-1ae6-4ba5-bf1e-14f15b509f23","Type":"ContainerDied","Data":"ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c"}
Jul 13 19:09:25.591295 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.591275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r" event={"ID":"d41206af-1ae6-4ba5-bf1e-14f15b509f23","Type":"ContainerDied","Data":"c4bb1a71f6c0b07166ae7dc3cbdb43d9ff67e2af5520e98079ce124fa822065a"}
Jul 13 19:09:25.591418 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.591301    2569 scope.go:117] "RemoveContainer" containerID="46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a"
Jul 13 19:09:25.601992 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.601972    2569 scope.go:117] "RemoveContainer" containerID="ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c"
Jul 13 19:09:25.609852 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.609833    2569 scope.go:117] "RemoveContainer" containerID="270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291"
Jul 13 19:09:25.628355 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.628105    2569 scope.go:117] "RemoveContainer" containerID="46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a"
Jul 13 19:09:25.628787 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:09:25.628756    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a\": container with ID starting with 46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a not found: ID does not exist" containerID="46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a"
Jul 13 19:09:25.628898 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.628799    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a"} err="failed to get container status \"46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a\": rpc error: code = NotFound desc = could not find container \"46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a\": container with ID starting with 46c959a7d0605b46a1430e7c57ed0e5fa6ae56573e2987ce5adf1a83f866022a not found: ID does not exist"
Jul 13 19:09:25.628898 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.628824    2569 scope.go:117] "RemoveContainer" containerID="ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c"
Jul 13 19:09:25.629205 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:09:25.629181    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c\": container with ID starting with ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c not found: ID does not exist" containerID="ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c"
Jul 13 19:09:25.629280 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.629227    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c"} err="failed to get container status \"ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c\": rpc error: code = NotFound desc = could not find container \"ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c\": container with ID starting with ac9d18f6d4ec3d6a515aca1f747e98148c7f7fa821df70d0cae571e9b6e72c0c not found: ID does not exist"
Jul 13 19:09:25.629280 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.629247    2569 scope.go:117] "RemoveContainer" containerID="270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291"
Jul 13 19:09:25.629635 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:09:25.629609    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291\": container with ID starting with 270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291 not found: ID does not exist" containerID="270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291"
Jul 13 19:09:25.629715 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.629652    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291"} err="failed to get container status \"270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291\": rpc error: code = NotFound desc = could not find container \"270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291\": container with ID starting with 270c0f2fa11e469987d2d1adc27e1ced003f654f4af07b44430cf498aa95e291 not found: ID does not exist"
Jul 13 19:09:25.630019 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.629997    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"]
Jul 13 19:09:25.635228 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:25.635205    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b1b9c-predictor-75fb4df589-hr65r"]
Jul 13 19:09:26.044787 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:26.044749    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" path="/var/lib/kubelet/pods/d41206af-1ae6-4ba5-bf1e-14f15b509f23/volumes"
Jul 13 19:09:26.597449 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:26.597418    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" event={"ID":"727c813c-bf1b-465a-ae4a-b37a12240845","Type":"ContainerStarted","Data":"cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec"}
Jul 13 19:09:26.597886 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:26.597474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" event={"ID":"727c813c-bf1b-465a-ae4a-b37a12240845","Type":"ContainerStarted","Data":"9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e"}
Jul 13 19:09:26.597886 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:26.597773    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:26.598004 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:26.597945    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:26.599045 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:26.599014    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:09:26.620627 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:26.620583    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podStartSLOduration=2.830390884 podStartE2EDuration="6.620568819s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.790177935s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:09:26.619949926 +0000 UTC m=+1158.725553567" watchObservedRunningTime="2026-07-13 19:09:26.620568819 +0000 UTC m=+1158.726172438"
Jul 13 19:09:27.600932 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:27.600887    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:09:28.605059 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:28.605018    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:09:33.610174 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:33.610098    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:09:33.610720 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:33.610691    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:09:43.610906 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:43.610864    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:09:53.611487 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:09:53.611439    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:10:03.611224 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:03.611182    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:10:09.951023 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:09.950995    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 19:10:09.951441 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:09.951079    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 19:10:13.611039 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:13.611000    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:10:23.611146 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:23.611107    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:10:33.611539 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:33.611492    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:10:38.041405 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:38.041366    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 13 19:10:48.045325 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:48.045291    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:10:50.867351 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.867320    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"]
Jul 13 19:10:50.867745 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.867628    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="storage-initializer"
Jul 13 19:10:50.867745 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.867641    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="storage-initializer"
Jul 13 19:10:50.867745 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.867658    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kube-rbac-proxy"
Jul 13 19:10:50.867745 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.867664    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kube-rbac-proxy"
Jul 13 19:10:50.867745 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.867673    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container"
Jul 13 19:10:50.867745 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.867678    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container"
Jul 13 19:10:50.867745 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.867723    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kserve-container"
Jul 13 19:10:50.867745 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.867731    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d41206af-1ae6-4ba5-bf1e-14f15b509f23" containerName="kube-rbac-proxy"
Jul 13 19:10:50.870712 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.870695    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:50.873439 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.873417    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-985ff8-kube-rbac-proxy-sar-config\""
Jul 13 19:10:50.873563 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.873417    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-985ff8\""
Jul 13 19:10:50.873563 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.873556    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-985ff8-dockercfg-pwcn4\""
Jul 13 19:10:50.874612 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.874598    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-985ff8-predictor-serving-cert\""
Jul 13 19:10:50.886413 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.886393    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"]
Jul 13 19:10:50.922776 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.922751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kserve-provision-location\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:50.922912 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.922794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:50.922912 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.922817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl795\" (UniqueName: \"kubernetes.io/projected/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-api-access-tl795\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:50.922912 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.922844    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-proxy-tls\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:50.922912 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.922874    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-cabundle-cert\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:50.992666 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:50.992639    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:10:51.024069 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.024038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kserve-provision-location\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.024215 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.024085    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.024215 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.024109    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl795\" (UniqueName: \"kubernetes.io/projected/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-api-access-tl795\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.024215 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.024141    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-proxy-tls\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.024215 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.024186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-cabundle-cert\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.024932 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.024906    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kserve-provision-location\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.025477 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.025445    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-cabundle-cert\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.025668 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.025477    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.028061 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.028036    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-proxy-tls\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.033330 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.033307    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl795\" (UniqueName: \"kubernetes.io/projected/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-api-access-tl795\") pod \"isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") " pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.181690 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.181657    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:10:51.309666 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.309637    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"]
Jul 13 19:10:51.312202 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:10:51.312164    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b86ec4_1a4b_4681_89c4_c3c24b4238ae.slice/crio-51c3769786c45f2caf6ff2606448fc02000e8824d27a74af65a604e119cd1818 WatchSource:0}: Error finding container 51c3769786c45f2caf6ff2606448fc02000e8824d27a74af65a604e119cd1818: Status 404 returned error can't find the container with id 51c3769786c45f2caf6ff2606448fc02000e8824d27a74af65a604e119cd1818
Jul 13 19:10:51.917829 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.917789    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8" event={"ID":"88b86ec4-1a4b-4681-89c4-c3c24b4238ae","Type":"ContainerStarted","Data":"d4a922e916f584bff7b13844fe00c2b223b7198cd8ad962ccdf37c0e57d885e5"}
Jul 13 19:10:51.918331 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:51.917848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8" event={"ID":"88b86ec4-1a4b-4681-89c4-c3c24b4238ae","Type":"ContainerStarted","Data":"51c3769786c45f2caf6ff2606448fc02000e8824d27a74af65a604e119cd1818"}
Jul 13 19:10:52.766700 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:52.766667    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:10:56.938027 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:56.938000    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8_88b86ec4-1a4b-4681-89c4-c3c24b4238ae/storage-initializer/0.log"
Jul 13 19:10:56.938419 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:56.938038    2569 generic.go:356] "Generic (PLEG): container finished" podID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae" containerID="d4a922e916f584bff7b13844fe00c2b223b7198cd8ad962ccdf37c0e57d885e5" exitCode=1
Jul 13 19:10:56.938419 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:56.938074    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8" event={"ID":"88b86ec4-1a4b-4681-89c4-c3c24b4238ae","Type":"ContainerDied","Data":"d4a922e916f584bff7b13844fe00c2b223b7198cd8ad962ccdf37c0e57d885e5"}
Jul 13 19:10:57.943792 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:57.943766    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8_88b86ec4-1a4b-4681-89c4-c3c24b4238ae/storage-initializer/0.log"
Jul 13 19:10:57.944210 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:10:57.943835    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8" event={"ID":"88b86ec4-1a4b-4681-89c4-c3c24b4238ae","Type":"ContainerStarted","Data":"b0690ac7edcb5cf1f3cb88520023db290877e79185dd6d76daf93a31b053eb2f"}
Jul 13 19:11:00.962155 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:00.962110    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8_88b86ec4-1a4b-4681-89c4-c3c24b4238ae/storage-initializer/1.log"
Jul 13 19:11:00.962573 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:00.962555    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8_88b86ec4-1a4b-4681-89c4-c3c24b4238ae/storage-initializer/0.log"
Jul 13 19:11:00.962637 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:00.962591    2569 generic.go:356] "Generic (PLEG): container finished" podID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae" containerID="b0690ac7edcb5cf1f3cb88520023db290877e79185dd6d76daf93a31b053eb2f" exitCode=1
Jul 13 19:11:00.962685 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:00.962665    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8" event={"ID":"88b86ec4-1a4b-4681-89c4-c3c24b4238ae","Type":"ContainerDied","Data":"b0690ac7edcb5cf1f3cb88520023db290877e79185dd6d76daf93a31b053eb2f"}
Jul 13 19:11:00.962738 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:00.962727    2569 scope.go:117] "RemoveContainer" containerID="d4a922e916f584bff7b13844fe00c2b223b7198cd8ad962ccdf37c0e57d885e5"
Jul 13 19:11:00.963094 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:00.963072    2569 scope.go:117] "RemoveContainer" containerID="d4a922e916f584bff7b13844fe00c2b223b7198cd8ad962ccdf37c0e57d885e5"
Jul 13 19:11:00.973441 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:11:00.973411    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8_kserve-ci-e2e-test(88b86ec4-1a4b-4681-89c4-c3c24b4238ae)\"" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8" podUID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae"
Jul 13 19:11:00.973581 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:11:00.973524    2569 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8_kserve-ci-e2e-test_88b86ec4-1a4b-4681-89c4-c3c24b4238ae_0 in pod sandbox 51c3769786c45f2caf6ff2606448fc02000e8824d27a74af65a604e119cd1818 from index: no such id: 'd4a922e916f584bff7b13844fe00c2b223b7198cd8ad962ccdf37c0e57d885e5'" containerID="d4a922e916f584bff7b13844fe00c2b223b7198cd8ad962ccdf37c0e57d885e5"
Jul 13 19:11:00.973581 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:00.973570    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a922e916f584bff7b13844fe00c2b223b7198cd8ad962ccdf37c0e57d885e5"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8_kserve-ci-e2e-test_88b86ec4-1a4b-4681-89c4-c3c24b4238ae_0 in pod sandbox 51c3769786c45f2caf6ff2606448fc02000e8824d27a74af65a604e119cd1818 from index: no such id: 'd4a922e916f584bff7b13844fe00c2b223b7198cd8ad962ccdf37c0e57d885e5'"
Jul 13 19:11:01.967693 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:01.967666    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8_88b86ec4-1a4b-4681-89c4-c3c24b4238ae/storage-initializer/1.log"
Jul 13 19:11:09.087637 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.087609    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"]
Jul 13 19:11:09.109902 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.109871    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:09.140114 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.140085    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"]
Jul 13 19:11:09.140379 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.140348    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container" containerID="cri-o://9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e" gracePeriod=30
Jul 13 19:11:09.140611 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.140387    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kube-rbac-proxy" containerID="cri-o://cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec" gracePeriod=30
Jul 13 19:11:09.231947 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.231910    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:09.259455 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.259431    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8_88b86ec4-1a4b-4681-89c4-c3c24b4238ae/storage-initializer/1.log"
Jul 13 19:11:09.261380 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.261365    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:11:09.379213 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.379134    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-rbac-proxy-sar-config\") pod \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") "
Jul 13 19:11:09.379213 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.379184    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl795\" (UniqueName: \"kubernetes.io/projected/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-api-access-tl795\") pod \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") "
Jul 13 19:11:09.379426 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.379228    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kserve-provision-location\") pod \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") "
Jul 13 19:11:09.379426 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.379250    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-proxy-tls\") pod \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") "
Jul 13 19:11:09.379426 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.379289    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-cabundle-cert\") pod \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\" (UID: \"88b86ec4-1a4b-4681-89c4-c3c24b4238ae\") "
Jul 13 19:11:09.379625 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.379514    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88b86ec4-1a4b-4681-89c4-c3c24b4238ae" (UID: "88b86ec4-1a4b-4681-89c4-c3c24b4238ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:09.379727 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.379701    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88b86ec4-1a4b-4681-89c4-c3c24b4238ae" (UID: "88b86ec4-1a4b-4681-89c4-c3c24b4238ae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:11:09.379796 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.379772    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88b86ec4-1a4b-4681-89c4-c3c24b4238ae" (UID: "88b86ec4-1a4b-4681-89c4-c3c24b4238ae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:11:09.381374 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.381347    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88b86ec4-1a4b-4681-89c4-c3c24b4238ae" (UID: "88b86ec4-1a4b-4681-89c4-c3c24b4238ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:11:09.381374 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.381358    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-api-access-tl795" (OuterVolumeSpecName: "kube-api-access-tl795") pod "88b86ec4-1a4b-4681-89c4-c3c24b4238ae" (UID: "88b86ec4-1a4b-4681-89c4-c3c24b4238ae"). InnerVolumeSpecName "kube-api-access-tl795". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:11:09.480447 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.480411    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl795\" (UniqueName: \"kubernetes.io/projected/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-api-access-tl795\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:09.480447 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.480435    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:09.480447 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.480445    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:09.480447 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.480453    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:09.480755 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:09.480462    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b86ec4-1a4b-4681-89c4-c3c24b4238ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:10.002937 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:10.002905    2569 generic.go:356] "Generic (PLEG): container finished" podID="727c813c-bf1b-465a-ae4a-b37a12240845" containerID="cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec" exitCode=2
Jul 13 19:11:10.003093 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:10.002978    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" event={"ID":"727c813c-bf1b-465a-ae4a-b37a12240845","Type":"ContainerDied","Data":"cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec"}
Jul 13 19:11:10.004185 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:10.004157    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8" event={"ID":"88b86ec4-1a4b-4681-89c4-c3c24b4238ae","Type":"ContainerDied","Data":"51c3769786c45f2caf6ff2606448fc02000e8824d27a74af65a604e119cd1818"}
Jul 13 19:11:10.004319 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:10.004192    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"
Jul 13 19:11:10.004319 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:10.004223    2569 scope.go:117] "RemoveContainer" containerID="b0690ac7edcb5cf1f3cb88520023db290877e79185dd6d76daf93a31b053eb2f"
Jul 13 19:11:10.059387 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:10.059349    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"]
Jul 13 19:11:10.063058 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:10.063030    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-985ff8-predictor-8dbc6b87c-9jwt8"]
Jul 13 19:11:10.307560 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:10.307457    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:12.045505 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:12.045473    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae" path="/var/lib/kubelet/pods/88b86ec4-1a4b-4681-89c4-c3c24b4238ae/volumes"
Jul 13 19:11:13.578994 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.578973    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:11:13.717169 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.717090    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727c813c-bf1b-465a-ae4a-b37a12240845-kserve-provision-location\") pod \"727c813c-bf1b-465a-ae4a-b37a12240845\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") "
Jul 13 19:11:13.717169 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.717140    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s49s5\" (UniqueName: \"kubernetes.io/projected/727c813c-bf1b-465a-ae4a-b37a12240845-kube-api-access-s49s5\") pod \"727c813c-bf1b-465a-ae4a-b37a12240845\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") "
Jul 13 19:11:13.717332 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.717183    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/727c813c-bf1b-465a-ae4a-b37a12240845-proxy-tls\") pod \"727c813c-bf1b-465a-ae4a-b37a12240845\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") "
Jul 13 19:11:13.717332 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.717205    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-kube-rbac-proxy-sar-config\") pod \"727c813c-bf1b-465a-ae4a-b37a12240845\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") "
Jul 13 19:11:13.717332 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.717247    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-cabundle-cert\") pod \"727c813c-bf1b-465a-ae4a-b37a12240845\" (UID: \"727c813c-bf1b-465a-ae4a-b37a12240845\") "
Jul 13 19:11:13.717544 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.717423    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727c813c-bf1b-465a-ae4a-b37a12240845-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "727c813c-bf1b-465a-ae4a-b37a12240845" (UID: "727c813c-bf1b-465a-ae4a-b37a12240845"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:13.717715 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.717620    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "727c813c-bf1b-465a-ae4a-b37a12240845" (UID: "727c813c-bf1b-465a-ae4a-b37a12240845"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:11:13.717715 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.717635    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "727c813c-bf1b-465a-ae4a-b37a12240845" (UID: "727c813c-bf1b-465a-ae4a-b37a12240845"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:11:13.719287 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.719262    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727c813c-bf1b-465a-ae4a-b37a12240845-kube-api-access-s49s5" (OuterVolumeSpecName: "kube-api-access-s49s5") pod "727c813c-bf1b-465a-ae4a-b37a12240845" (UID: "727c813c-bf1b-465a-ae4a-b37a12240845"). InnerVolumeSpecName "kube-api-access-s49s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:11:13.719364 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.719291    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727c813c-bf1b-465a-ae4a-b37a12240845-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "727c813c-bf1b-465a-ae4a-b37a12240845" (UID: "727c813c-bf1b-465a-ae4a-b37a12240845"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:11:13.817968 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.817927    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:13.817968 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.817955    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727c813c-bf1b-465a-ae4a-b37a12240845-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:13.817968 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.817965    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s49s5\" (UniqueName: \"kubernetes.io/projected/727c813c-bf1b-465a-ae4a-b37a12240845-kube-api-access-s49s5\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:13.817968 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.817974    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/727c813c-bf1b-465a-ae4a-b37a12240845-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:13.818302 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:13.817984    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/727c813c-bf1b-465a-ae4a-b37a12240845-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:14.022733 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.022652    2569 generic.go:356] "Generic (PLEG): container finished" podID="727c813c-bf1b-465a-ae4a-b37a12240845" containerID="9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e" exitCode=0
Jul 13 19:11:14.022733 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.022699    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" event={"ID":"727c813c-bf1b-465a-ae4a-b37a12240845","Type":"ContainerDied","Data":"9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e"}
Jul 13 19:11:14.022733 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.022724    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"
Jul 13 19:11:14.022994 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.022746    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl" event={"ID":"727c813c-bf1b-465a-ae4a-b37a12240845","Type":"ContainerDied","Data":"bd6445acf6a70952c78edbf13c553e33f4d604317a1dbe1bff9df27063cb0d62"}
Jul 13 19:11:14.022994 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.022771    2569 scope.go:117] "RemoveContainer" containerID="cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec"
Jul 13 19:11:14.030876 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.030859    2569 scope.go:117] "RemoveContainer" containerID="9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e"
Jul 13 19:11:14.038551 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.038512    2569 scope.go:117] "RemoveContainer" containerID="c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22"
Jul 13 19:11:14.046581 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.046558    2569 scope.go:117] "RemoveContainer" containerID="cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec"
Jul 13 19:11:14.046874 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:11:14.046850    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec\": container with ID starting with cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec not found: ID does not exist" containerID="cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec"
Jul 13 19:11:14.046943 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.046883    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec"} err="failed to get container status \"cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec\": rpc error: code = NotFound desc = could not find container \"cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec\": container with ID starting with cf22ebd38b5e606798167955634f4527e5e6bc91adecb7afa495d9f63d01f1ec not found: ID does not exist"
Jul 13 19:11:14.046943 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.046902    2569 scope.go:117] "RemoveContainer" containerID="9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e"
Jul 13 19:11:14.047184 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:11:14.047147    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e\": container with ID starting with 9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e not found: ID does not exist" containerID="9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e"
Jul 13 19:11:14.047240 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.047190    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e"} err="failed to get container status \"9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e\": rpc error: code = NotFound desc = could not find container \"9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e\": container with ID starting with 9bee9e8a7d0e01a728f7992d89ec7efca13d4f8364b1b2e568a8619a96132a7e not found: ID does not exist"
Jul 13 19:11:14.047240 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.047206    2569 scope.go:117] "RemoveContainer" containerID="c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22"
Jul 13 19:11:14.047487 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:11:14.047460    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22\": container with ID starting with c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22 not found: ID does not exist" containerID="c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22"
Jul 13 19:11:14.047558 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.047492    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22"} err="failed to get container status \"c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22\": rpc error: code = NotFound desc = could not find container \"c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22\": container with ID starting with c1f2c1f95bcc486cc336dde738bb2bd5820e7c790a54bab2f09c604453ab6c22 not found: ID does not exist"
Jul 13 19:11:14.052388 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.052362    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"]
Jul 13 19:11:14.057970 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.057947    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-985ff8-predictor-67fb957c74-z66gl"]
Jul 13 19:11:14.183992 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.183960    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:14.253976 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.253941    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"]
Jul 13 19:11:14.254305 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254289    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae" containerName="storage-initializer"
Jul 13 19:11:14.254388 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254309    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae" containerName="storage-initializer"
Jul 13 19:11:14.254388 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254345    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kube-rbac-proxy"
Jul 13 19:11:14.254388 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254355    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kube-rbac-proxy"
Jul 13 19:11:14.254388 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254377    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae" containerName="storage-initializer"
Jul 13 19:11:14.254388 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254388    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae" containerName="storage-initializer"
Jul 13 19:11:14.254669 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254403    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="storage-initializer"
Jul 13 19:11:14.254669 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254414    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="storage-initializer"
Jul 13 19:11:14.254669 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254425    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container"
Jul 13 19:11:14.254669 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254434    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container"
Jul 13 19:11:14.254669 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254502    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kube-rbac-proxy"
Jul 13 19:11:14.254669 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254515    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae" containerName="storage-initializer"
Jul 13 19:11:14.254669 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254544    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" containerName="kserve-container"
Jul 13 19:11:14.255005 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.254675    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b86ec4-1a4b-4681-89c4-c3c24b4238ae" containerName="storage-initializer"
Jul 13 19:11:14.259113 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.259092    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.265093 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.265065    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:11:14.265641 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.265627    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6a2fcd-kube-rbac-proxy-sar-config\""
Jul 13 19:11:14.266250 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.265834    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6a2fcd-predictor-serving-cert\""
Jul 13 19:11:14.266250 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.265960    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-6a2fcd-dockercfg-smlf6\""
Jul 13 19:11:14.266250 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.266113    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 19:11:14.266250 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.266196    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:11:14.266461 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.266376    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-6a2fcd\""
Jul 13 19:11:14.282820 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.282723    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"]
Jul 13 19:11:14.322490 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.322452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8kkd\" (UniqueName: \"kubernetes.io/projected/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-api-access-w8kkd\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.322677 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.322495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/202e6c20-cf3e-49e0-89f4-22a2d426498f-proxy-tls\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.322677 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.322612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.322677 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.322642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e6c20-cf3e-49e0-89f4-22a2d426498f-kserve-provision-location\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.322677 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.322659    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-cabundle-cert\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.395362 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.395330    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:14.423470 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.423437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8kkd\" (UniqueName: \"kubernetes.io/projected/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-api-access-w8kkd\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.423609 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.423479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/202e6c20-cf3e-49e0-89f4-22a2d426498f-proxy-tls\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.423609 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.423524    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.423609 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.423573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e6c20-cf3e-49e0-89f4-22a2d426498f-kserve-provision-location\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.423609 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.423593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-cabundle-cert\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.424029 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.424005    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e6c20-cf3e-49e0-89f4-22a2d426498f-kserve-provision-location\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.424275 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.424259    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.424366 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.424267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-cabundle-cert\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.425936 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.425917    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/202e6c20-cf3e-49e0-89f4-22a2d426498f-proxy-tls\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.433251 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.433224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8kkd\" (UniqueName: \"kubernetes.io/projected/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-api-access-w8kkd\") pod \"isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.569403 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.569308    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:14.729506 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:14.729468    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"]
Jul 13 19:11:14.732597 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:11:14.732565    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202e6c20_cf3e_49e0_89f4_22a2d426498f.slice/crio-0ab0158c146f03ee8a0e33e782f68f7c595c05c315e21c79009e61a512dee33b WatchSource:0}: Error finding container 0ab0158c146f03ee8a0e33e782f68f7c595c05c315e21c79009e61a512dee33b: Status 404 returned error can't find the container with id 0ab0158c146f03ee8a0e33e782f68f7c595c05c315e21c79009e61a512dee33b
Jul 13 19:11:15.029783 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:15.029742    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz" event={"ID":"202e6c20-cf3e-49e0-89f4-22a2d426498f","Type":"ContainerStarted","Data":"b2810fa65c842757957874dca524186886a96b1d1c794aa7b08b456d3c953233"}
Jul 13 19:11:15.029954 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:15.029825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz" event={"ID":"202e6c20-cf3e-49e0-89f4-22a2d426498f","Type":"ContainerStarted","Data":"0ab0158c146f03ee8a0e33e782f68f7c595c05c315e21c79009e61a512dee33b"}
Jul 13 19:11:16.044581 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:16.044545    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727c813c-bf1b-465a-ae4a-b37a12240845" path="/var/lib/kubelet/pods/727c813c-bf1b-465a-ae4a-b37a12240845/volumes"
Jul 13 19:11:19.046413 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:19.046388    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_202e6c20-cf3e-49e0-89f4-22a2d426498f/storage-initializer/0.log"
Jul 13 19:11:19.046789 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:19.046425    2569 generic.go:356] "Generic (PLEG): container finished" podID="202e6c20-cf3e-49e0-89f4-22a2d426498f" containerID="b2810fa65c842757957874dca524186886a96b1d1c794aa7b08b456d3c953233" exitCode=1
Jul 13 19:11:19.046789 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:19.046459    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz" event={"ID":"202e6c20-cf3e-49e0-89f4-22a2d426498f","Type":"ContainerDied","Data":"b2810fa65c842757957874dca524186886a96b1d1c794aa7b08b456d3c953233"}
Jul 13 19:11:20.057539 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:20.057496    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_202e6c20-cf3e-49e0-89f4-22a2d426498f/storage-initializer/0.log"
Jul 13 19:11:20.057923 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:20.057568    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz" event={"ID":"202e6c20-cf3e-49e0-89f4-22a2d426498f","Type":"ContainerStarted","Data":"cdd23ae851fc05aa123d5c0a3ffb504c8d0ad656badb93a5e74a7c0d5d22544c"}
Jul 13 19:11:23.069812 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:23.069782    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_202e6c20-cf3e-49e0-89f4-22a2d426498f/storage-initializer/1.log"
Jul 13 19:11:23.070286 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:23.070160    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_202e6c20-cf3e-49e0-89f4-22a2d426498f/storage-initializer/0.log"
Jul 13 19:11:23.070286 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:23.070191    2569 generic.go:356] "Generic (PLEG): container finished" podID="202e6c20-cf3e-49e0-89f4-22a2d426498f" containerID="cdd23ae851fc05aa123d5c0a3ffb504c8d0ad656badb93a5e74a7c0d5d22544c" exitCode=1
Jul 13 19:11:23.070286 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:23.070235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz" event={"ID":"202e6c20-cf3e-49e0-89f4-22a2d426498f","Type":"ContainerDied","Data":"cdd23ae851fc05aa123d5c0a3ffb504c8d0ad656badb93a5e74a7c0d5d22544c"}
Jul 13 19:11:23.070390 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:23.070297    2569 scope.go:117] "RemoveContainer" containerID="b2810fa65c842757957874dca524186886a96b1d1c794aa7b08b456d3c953233"
Jul 13 19:11:23.070725 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:23.070701    2569 scope.go:117] "RemoveContainer" containerID="b2810fa65c842757957874dca524186886a96b1d1c794aa7b08b456d3c953233"
Jul 13 19:11:23.080857 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:11:23.080833    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_kserve-ci-e2e-test(202e6c20-cf3e-49e0-89f4-22a2d426498f)\"" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz" podUID="202e6c20-cf3e-49e0-89f4-22a2d426498f"
Jul 13 19:11:23.080928 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:11:23.080905    2569 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_kserve-ci-e2e-test_202e6c20-cf3e-49e0-89f4-22a2d426498f_0 in pod sandbox 0ab0158c146f03ee8a0e33e782f68f7c595c05c315e21c79009e61a512dee33b from index: no such id: 'b2810fa65c842757957874dca524186886a96b1d1c794aa7b08b456d3c953233'" containerID="b2810fa65c842757957874dca524186886a96b1d1c794aa7b08b456d3c953233"
Jul 13 19:11:23.080974 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:23.080937    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2810fa65c842757957874dca524186886a96b1d1c794aa7b08b456d3c953233"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_kserve-ci-e2e-test_202e6c20-cf3e-49e0-89f4-22a2d426498f_0 in pod sandbox 0ab0158c146f03ee8a0e33e782f68f7c595c05c315e21c79009e61a512dee33b from index: no such id: 'b2810fa65c842757957874dca524186886a96b1d1c794aa7b08b456d3c953233'"
Jul 13 19:11:24.075854 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.075821    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_202e6c20-cf3e-49e0-89f4-22a2d426498f/storage-initializer/1.log"
Jul 13 19:11:24.200810 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.200780    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"]
Jul 13 19:11:24.339033 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.339012    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_202e6c20-cf3e-49e0-89f4-22a2d426498f/storage-initializer/1.log"
Jul 13 19:11:24.342653 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.342637    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:24.504437 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.504392    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e6c20-cf3e-49e0-89f4-22a2d426498f-kserve-provision-location\") pod \"202e6c20-cf3e-49e0-89f4-22a2d426498f\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") "
Jul 13 19:11:24.504641 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.504465    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-rbac-proxy-sar-config\") pod \"202e6c20-cf3e-49e0-89f4-22a2d426498f\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") "
Jul 13 19:11:24.504641 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.504518    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-cabundle-cert\") pod \"202e6c20-cf3e-49e0-89f4-22a2d426498f\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") "
Jul 13 19:11:24.504641 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.504566    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/202e6c20-cf3e-49e0-89f4-22a2d426498f-proxy-tls\") pod \"202e6c20-cf3e-49e0-89f4-22a2d426498f\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") "
Jul 13 19:11:24.504641 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.504597    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8kkd\" (UniqueName: \"kubernetes.io/projected/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-api-access-w8kkd\") pod \"202e6c20-cf3e-49e0-89f4-22a2d426498f\" (UID: \"202e6c20-cf3e-49e0-89f4-22a2d426498f\") "
Jul 13 19:11:24.504867 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.504671    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202e6c20-cf3e-49e0-89f4-22a2d426498f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "202e6c20-cf3e-49e0-89f4-22a2d426498f" (UID: "202e6c20-cf3e-49e0-89f4-22a2d426498f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:24.504867 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.504803    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e6c20-cf3e-49e0-89f4-22a2d426498f-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:24.504867 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.504851    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "202e6c20-cf3e-49e0-89f4-22a2d426498f" (UID: "202e6c20-cf3e-49e0-89f4-22a2d426498f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:11:24.505017 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.504951    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "202e6c20-cf3e-49e0-89f4-22a2d426498f" (UID: "202e6c20-cf3e-49e0-89f4-22a2d426498f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:11:24.506705 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.506684    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-api-access-w8kkd" (OuterVolumeSpecName: "kube-api-access-w8kkd") pod "202e6c20-cf3e-49e0-89f4-22a2d426498f" (UID: "202e6c20-cf3e-49e0-89f4-22a2d426498f"). InnerVolumeSpecName "kube-api-access-w8kkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:11:24.506775 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.506684    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202e6c20-cf3e-49e0-89f4-22a2d426498f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "202e6c20-cf3e-49e0-89f4-22a2d426498f" (UID: "202e6c20-cf3e-49e0-89f4-22a2d426498f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:11:24.605225 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.605151    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:24.605225 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.605177    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/202e6c20-cf3e-49e0-89f4-22a2d426498f-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:24.605225 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.605185    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8kkd\" (UniqueName: \"kubernetes.io/projected/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-api-access-w8kkd\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:24.605225 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:24.605196    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/202e6c20-cf3e-49e0-89f4-22a2d426498f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:11:25.080519 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:25.080491    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz_202e6c20-cf3e-49e0-89f4-22a2d426498f/storage-initializer/1.log"
Jul 13 19:11:25.080935 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:25.080589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz" event={"ID":"202e6c20-cf3e-49e0-89f4-22a2d426498f","Type":"ContainerDied","Data":"0ab0158c146f03ee8a0e33e782f68f7c595c05c315e21c79009e61a512dee33b"}
Jul 13 19:11:25.080935 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:25.080604    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"
Jul 13 19:11:25.080935 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:25.080638    2569 scope.go:117] "RemoveContainer" containerID="cdd23ae851fc05aa123d5c0a3ffb504c8d0ad656badb93a5e74a7c0d5d22544c"
Jul 13 19:11:25.133902 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:25.133871    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"]
Jul 13 19:11:25.135874 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:25.135850    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6a2fcd-predictor-7b7df4594c-2qktz"]
Jul 13 19:11:25.568222 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:25.568187    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:26.045114 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:26.045076    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="202e6c20-cf3e-49e0-89f4-22a2d426498f" path="/var/lib/kubelet/pods/202e6c20-cf3e-49e0-89f4-22a2d426498f/volumes"
Jul 13 19:11:29.522170 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.522138    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:29.582357 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.582329    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"]
Jul 13 19:11:29.582620 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.582608    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202e6c20-cf3e-49e0-89f4-22a2d426498f" containerName="storage-initializer"
Jul 13 19:11:29.582662 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.582622    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="202e6c20-cf3e-49e0-89f4-22a2d426498f" containerName="storage-initializer"
Jul 13 19:11:29.582662 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.582633    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202e6c20-cf3e-49e0-89f4-22a2d426498f" containerName="storage-initializer"
Jul 13 19:11:29.582662 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.582639    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="202e6c20-cf3e-49e0-89f4-22a2d426498f" containerName="storage-initializer"
Jul 13 19:11:29.582755 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.582691    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="202e6c20-cf3e-49e0-89f4-22a2d426498f" containerName="storage-initializer"
Jul 13 19:11:29.582755 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.582701    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="202e6c20-cf3e-49e0-89f4-22a2d426498f" containerName="storage-initializer"
Jul 13 19:11:29.586736 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.586716    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.593312 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.593290    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:11:29.593404 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.593355    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-f1f2c-kube-rbac-proxy-sar-config\""
Jul 13 19:11:29.593541 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.593510    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 19:11:29.594509 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.594487    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mgp78\""
Jul 13 19:11:29.594627 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.594492    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:11:29.594627 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.594549    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:11:29.594627 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.594570    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-f1f2c-predictor-serving-cert\""
Jul 13 19:11:29.617070 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.617038    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"]
Jul 13 19:11:29.693459 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.693426    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:11:29.741667 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.741632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.741667 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.741669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kserve-provision-location\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.741835 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.741744    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9b6e7e1-e413-42e0-84df-4463dd0d9686-proxy-tls\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.741835 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.741773    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-cabundle-cert\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.741835 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.741819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5vd7\" (UniqueName: \"kubernetes.io/projected/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-api-access-s5vd7\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.842542 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.842440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.842542 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.842476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kserve-provision-location\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.842542 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.842521    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9b6e7e1-e413-42e0-84df-4463dd0d9686-proxy-tls\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.842799 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.842625    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-cabundle-cert\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.842799 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.842673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5vd7\" (UniqueName: \"kubernetes.io/projected/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-api-access-s5vd7\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.843037 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.843017    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kserve-provision-location\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.843165 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.843146    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.843232 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.843209    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-cabundle-cert\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.844966 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.844945    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9b6e7e1-e413-42e0-84df-4463dd0d9686-proxy-tls\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.851571 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.851552    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5vd7\" (UniqueName: \"kubernetes.io/projected/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-api-access-s5vd7\") pod \"raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") " pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:29.896835 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:29.896805    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:30.034777 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:30.034722    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"]
Jul 13 19:11:30.037655 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:11:30.037626    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b6e7e1_e413_42e0_84df_4463dd0d9686.slice/crio-1e0e57e1afdc216258bd9bb9ec826d456e7af1bc2130472970a0426f4c73082f WatchSource:0}: Error finding container 1e0e57e1afdc216258bd9bb9ec826d456e7af1bc2130472970a0426f4c73082f: Status 404 returned error can't find the container with id 1e0e57e1afdc216258bd9bb9ec826d456e7af1bc2130472970a0426f4c73082f
Jul 13 19:11:30.101409 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:30.101341    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" event={"ID":"c9b6e7e1-e413-42e0-84df-4463dd0d9686","Type":"ContainerStarted","Data":"99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e"}
Jul 13 19:11:30.101524 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:30.101407    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" event={"ID":"c9b6e7e1-e413-42e0-84df-4463dd0d9686","Type":"ContainerStarted","Data":"1e0e57e1afdc216258bd9bb9ec826d456e7af1bc2130472970a0426f4c73082f"}
Jul 13 19:11:34.116734 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:34.116652    2569 generic.go:356] "Generic (PLEG): container finished" podID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerID="99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e" exitCode=0
Jul 13 19:11:34.116734 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:34.116719    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" event={"ID":"c9b6e7e1-e413-42e0-84df-4463dd0d9686","Type":"ContainerDied","Data":"99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e"}
Jul 13 19:11:35.123078 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:35.123045    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" event={"ID":"c9b6e7e1-e413-42e0-84df-4463dd0d9686","Type":"ContainerStarted","Data":"09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879"}
Jul 13 19:11:35.123454 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:35.123105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" event={"ID":"c9b6e7e1-e413-42e0-84df-4463dd0d9686","Type":"ContainerStarted","Data":"5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7"}
Jul 13 19:11:35.123454 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:35.123370    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:35.123550 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:35.123521    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:35.124758 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:35.124731    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:11:36.132454 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:36.132416    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:11:37.136212 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:37.136166    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:11:42.141550 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:42.141495    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:11:42.142061 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:42.142034    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:11:42.193459 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:42.193407    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podStartSLOduration=9.550983331 podStartE2EDuration="13.193392204s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.642408873s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:11:35.149340903 +0000 UTC m=+1287.254944532" watchObservedRunningTime="2026-07-13 19:11:42.193392204 +0000 UTC m=+1294.298995884"
Jul 13 19:11:52.142456 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:11:52.142420    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:12:02.142201 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:02.142159    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:12:12.142578 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:12.142522    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:12:21.920852 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:21.920819    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:12:22.142852 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:22.142814    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:12:32.142458 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:32.142369    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:12:42.142351 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:42.142313    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:12:52.143321 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:52.143291    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:12:59.554417 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:59.554384    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"]
Jul 13 19:12:59.554889 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:59.554633    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" containerID="cri-o://5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7" gracePeriod=30
Jul 13 19:12:59.554889 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:59.554724    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kube-rbac-proxy" containerID="cri-o://09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879" gracePeriod=30
Jul 13 19:12:59.579355 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:12:59.579330    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:00.449363 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:00.449332    2569 generic.go:356] "Generic (PLEG): container finished" podID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerID="09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879" exitCode=2
Jul 13 19:13:00.449555 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:00.449414    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" event={"ID":"c9b6e7e1-e413-42e0-84df-4463dd0d9686","Type":"ContainerDied","Data":"09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879"}
Jul 13 19:13:02.136880 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:02.136838    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 13 19:13:02.142212 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:02.142188    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 13 19:13:04.201919 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.201896    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:13:04.300737 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.300661    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-rbac-proxy-sar-config\") pod \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") "
Jul 13 19:13:04.300737 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.300710    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kserve-provision-location\") pod \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") "
Jul 13 19:13:04.300737 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.300727    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5vd7\" (UniqueName: \"kubernetes.io/projected/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-api-access-s5vd7\") pod \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") "
Jul 13 19:13:04.300970 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.300770    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-cabundle-cert\") pod \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") "
Jul 13 19:13:04.300970 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.300810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9b6e7e1-e413-42e0-84df-4463dd0d9686-proxy-tls\") pod \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\" (UID: \"c9b6e7e1-e413-42e0-84df-4463dd0d9686\") "
Jul 13 19:13:04.301072 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.301038    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c9b6e7e1-e413-42e0-84df-4463dd0d9686" (UID: "c9b6e7e1-e413-42e0-84df-4463dd0d9686"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:13:04.301117 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.301067    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c9b6e7e1-e413-42e0-84df-4463dd0d9686" (UID: "c9b6e7e1-e413-42e0-84df-4463dd0d9686"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:13:04.301117 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.301075    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c9b6e7e1-e413-42e0-84df-4463dd0d9686" (UID: "c9b6e7e1-e413-42e0-84df-4463dd0d9686"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:13:04.302908 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.302882    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-api-access-s5vd7" (OuterVolumeSpecName: "kube-api-access-s5vd7") pod "c9b6e7e1-e413-42e0-84df-4463dd0d9686" (UID: "c9b6e7e1-e413-42e0-84df-4463dd0d9686"). InnerVolumeSpecName "kube-api-access-s5vd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:13:04.303036 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.302926    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b6e7e1-e413-42e0-84df-4463dd0d9686-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c9b6e7e1-e413-42e0-84df-4463dd0d9686" (UID: "c9b6e7e1-e413-42e0-84df-4463dd0d9686"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:13:04.401923 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.401897    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:13:04.401923 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.401918    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9b6e7e1-e413-42e0-84df-4463dd0d9686-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:13:04.401923 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.401928    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:13:04.402125 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.401936    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:13:04.402125 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.401945    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5vd7\" (UniqueName: \"kubernetes.io/projected/c9b6e7e1-e413-42e0-84df-4463dd0d9686-kube-api-access-s5vd7\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:13:04.465439 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.465408    2569 generic.go:356] "Generic (PLEG): container finished" podID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerID="5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7" exitCode=0
Jul 13 19:13:04.465596 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.465477    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"
Jul 13 19:13:04.465596 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.465505    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" event={"ID":"c9b6e7e1-e413-42e0-84df-4463dd0d9686","Type":"ContainerDied","Data":"5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7"}
Jul 13 19:13:04.465596 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.465580    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7" event={"ID":"c9b6e7e1-e413-42e0-84df-4463dd0d9686","Type":"ContainerDied","Data":"1e0e57e1afdc216258bd9bb9ec826d456e7af1bc2130472970a0426f4c73082f"}
Jul 13 19:13:04.465702 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.465606    2569 scope.go:117] "RemoveContainer" containerID="09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879"
Jul 13 19:13:04.473477 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.473459    2569 scope.go:117] "RemoveContainer" containerID="5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7"
Jul 13 19:13:04.480599 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.480519    2569 scope.go:117] "RemoveContainer" containerID="99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e"
Jul 13 19:13:04.486887 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.486864    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"]
Jul 13 19:13:04.487705 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.487693    2569 scope.go:117] "RemoveContainer" containerID="09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879"
Jul 13 19:13:04.487936 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:13:04.487921    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879\": container with ID starting with 09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879 not found: ID does not exist" containerID="09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879"
Jul 13 19:13:04.487982 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.487943    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879"} err="failed to get container status \"09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879\": rpc error: code = NotFound desc = could not find container \"09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879\": container with ID starting with 09a459d129bc0cbca1f8eadfff89681d72ded04c0d963132d8896a3eed027879 not found: ID does not exist"
Jul 13 19:13:04.487982 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.487958    2569 scope.go:117] "RemoveContainer" containerID="5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7"
Jul 13 19:13:04.488190 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:13:04.488169    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7\": container with ID starting with 5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7 not found: ID does not exist" containerID="5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7"
Jul 13 19:13:04.488233 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.488196    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7"} err="failed to get container status \"5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7\": rpc error: code = NotFound desc = could not find container \"5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7\": container with ID starting with 5011dcbe2109ff06c951a45692a7ebf09a84ce6dd5f14a95e05d6bc401ac61c7 not found: ID does not exist"
Jul 13 19:13:04.488233 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.488211    2569 scope.go:117] "RemoveContainer" containerID="99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e"
Jul 13 19:13:04.488428 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:13:04.488411    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e\": container with ID starting with 99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e not found: ID does not exist" containerID="99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e"
Jul 13 19:13:04.488467 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.488432    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e"} err="failed to get container status \"99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e\": rpc error: code = NotFound desc = could not find container \"99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e\": container with ID starting with 99af628b637247eba92165d94e1078fa86097d7bdd08a6cb4b28662a1974685e not found: ID does not exist"
Jul 13 19:13:04.490942 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.490921    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-f1f2c-predictor-5c547d56ff-ffgs7"]
Jul 13 19:13:04.682268 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.682242    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:04.731323 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.731288    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"]
Jul 13 19:13:04.731676 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.731658    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container"
Jul 13 19:13:04.731773 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.731678    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container"
Jul 13 19:13:04.731773 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.731694    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="storage-initializer"
Jul 13 19:13:04.731773 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.731702    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="storage-initializer"
Jul 13 19:13:04.731773 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.731722    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kube-rbac-proxy"
Jul 13 19:13:04.731773 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.731733    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kube-rbac-proxy"
Jul 13 19:13:04.732037 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.731800    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kserve-container"
Jul 13 19:13:04.732037 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.731812    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" containerName="kube-rbac-proxy"
Jul 13 19:13:04.737391 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.737360    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.740263 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.740231    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:13:04.740263 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.740261    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mgp78\""
Jul 13 19:13:04.740457 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.740279    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-09af0-predictor-serving-cert\""
Jul 13 19:13:04.740457 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.740333    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:13:04.740636 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.740611    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-09af0-kube-rbac-proxy-sar-config\""
Jul 13 19:13:04.740742 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.740715    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:13:04.740805 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.740736    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 19:13:04.744016 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.743995    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"]
Jul 13 19:13:04.805858 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.805801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.805858 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.805859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d48667-72e8-463e-866a-ee00e4882c2a-kserve-provision-location\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.806094 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.805883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t87l5\" (UniqueName: \"kubernetes.io/projected/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-api-access-t87l5\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.806094 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.805903    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-cabundle-cert\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.806094 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.805930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d48667-72e8-463e-866a-ee00e4882c2a-proxy-tls\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.906574 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.906547    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-cabundle-cert\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.906724 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.906583    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d48667-72e8-463e-866a-ee00e4882c2a-proxy-tls\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.906724 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.906645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.906724 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.906661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d48667-72e8-463e-866a-ee00e4882c2a-kserve-provision-location\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.906871 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.906780    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t87l5\" (UniqueName: \"kubernetes.io/projected/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-api-access-t87l5\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.907091 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.907067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d48667-72e8-463e-866a-ee00e4882c2a-kserve-provision-location\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.907276 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.907255    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.907353 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.907273    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-cabundle-cert\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.909033 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.909012    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d48667-72e8-463e-866a-ee00e4882c2a-proxy-tls\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.915504 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.915484    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t87l5\" (UniqueName: \"kubernetes.io/projected/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-api-access-t87l5\") pod \"raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:04.961027 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:04.960966    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:13:05.048906 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:05.048881    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:05.177409 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:05.177380    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"]
Jul 13 19:13:05.179463 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:13:05.179438    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d48667_72e8_463e_866a_ee00e4882c2a.slice/crio-508cdd0942ee477b0a26865fda79e02d79f39ee2fd7f1b124e17b546593cd5a8 WatchSource:0}: Error finding container 508cdd0942ee477b0a26865fda79e02d79f39ee2fd7f1b124e17b546593cd5a8: Status 404 returned error can't find the container with id 508cdd0942ee477b0a26865fda79e02d79f39ee2fd7f1b124e17b546593cd5a8
Jul 13 19:13:05.470583 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:05.470465    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" event={"ID":"c2d48667-72e8-463e-866a-ee00e4882c2a","Type":"ContainerStarted","Data":"cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742"}
Jul 13 19:13:05.470583 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:05.470548    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" event={"ID":"c2d48667-72e8-463e-866a-ee00e4882c2a","Type":"ContainerStarted","Data":"508cdd0942ee477b0a26865fda79e02d79f39ee2fd7f1b124e17b546593cd5a8"}
Jul 13 19:13:06.044910 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:06.044876    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b6e7e1-e413-42e0-84df-4463dd0d9686" path="/var/lib/kubelet/pods/c9b6e7e1-e413-42e0-84df-4463dd0d9686/volumes"
Jul 13 19:13:09.489496 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:09.489459    2569 generic.go:356] "Generic (PLEG): container finished" podID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerID="cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742" exitCode=0
Jul 13 19:13:09.489905 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:09.489507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" event={"ID":"c2d48667-72e8-463e-866a-ee00e4882c2a","Type":"ContainerDied","Data":"cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742"}
Jul 13 19:13:10.495222 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:10.495188    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" event={"ID":"c2d48667-72e8-463e-866a-ee00e4882c2a","Type":"ContainerStarted","Data":"de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592"}
Jul 13 19:13:10.495655 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:10.495250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" event={"ID":"c2d48667-72e8-463e-866a-ee00e4882c2a","Type":"ContainerStarted","Data":"411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246"}
Jul 13 19:13:10.495655 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:10.495550    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:10.495655 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:10.495612    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:10.496959 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:10.496931    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:10.516192 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:10.516139    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podStartSLOduration=2.743079023 podStartE2EDuration="6.516125687s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.773046664s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:13:10.51409995 +0000 UTC m=+1382.619703594" watchObservedRunningTime="2026-07-13 19:13:10.516125687 +0000 UTC m=+1382.621729306"
Jul 13 19:13:11.499766 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:11.499718    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:16.504520 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:16.504487    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:13:16.505130 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:16.505107    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:26.505754 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:26.505710    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:36.505120 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:36.505083    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:46.505344 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:46.505289    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:13:56.505466 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:13:56.505420    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:14:02.630787 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:02.630682    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:14:06.505628 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:06.505580    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:14:16.505847 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:16.505805    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:14:17.041783 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:17.041748    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:14:27.043309 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:27.043280    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:14:34.929508 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:34.929472    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"]
Jul 13 19:14:34.930043 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:34.929708    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" containerID="cri-o://411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246" gracePeriod=30
Jul 13 19:14:34.930043 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:34.929768    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kube-rbac-proxy" containerID="cri-o://de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592" gracePeriod=30
Jul 13 19:14:35.700922 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:35.700880    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:14:35.807307 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:35.807275    2569 generic.go:356] "Generic (PLEG): container finished" podID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerID="de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592" exitCode=2
Jul 13 19:14:35.807460 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:35.807348    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" event={"ID":"c2d48667-72e8-463e-866a-ee00e4882c2a","Type":"ContainerDied","Data":"de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592"}
Jul 13 19:14:36.500183 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:36.500140    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 13 19:14:37.042752 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:37.042715    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 13 19:14:39.576151 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.576124    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:14:39.579781 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.579762    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-rbac-proxy-sar-config\") pod \"c2d48667-72e8-463e-866a-ee00e4882c2a\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") "
Jul 13 19:14:39.579844 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.579814    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d48667-72e8-463e-866a-ee00e4882c2a-proxy-tls\") pod \"c2d48667-72e8-463e-866a-ee00e4882c2a\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") "
Jul 13 19:14:39.579882 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.579857    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t87l5\" (UniqueName: \"kubernetes.io/projected/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-api-access-t87l5\") pod \"c2d48667-72e8-463e-866a-ee00e4882c2a\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") "
Jul 13 19:14:39.579916 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.579889    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d48667-72e8-463e-866a-ee00e4882c2a-kserve-provision-location\") pod \"c2d48667-72e8-463e-866a-ee00e4882c2a\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") "
Jul 13 19:14:39.579951 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.579919    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-cabundle-cert\") pod \"c2d48667-72e8-463e-866a-ee00e4882c2a\" (UID: \"c2d48667-72e8-463e-866a-ee00e4882c2a\") "
Jul 13 19:14:39.580036 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.580013    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2d48667-72e8-463e-866a-ee00e4882c2a" (UID: "c2d48667-72e8-463e-866a-ee00e4882c2a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:14:39.580219 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.580192    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d48667-72e8-463e-866a-ee00e4882c2a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2d48667-72e8-463e-866a-ee00e4882c2a" (UID: "c2d48667-72e8-463e-866a-ee00e4882c2a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:39.580319 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.580287    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d48667-72e8-463e-866a-ee00e4882c2a-kserve-provision-location\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:14:39.580319 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.580307    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:14:39.580410 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.580329    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c2d48667-72e8-463e-866a-ee00e4882c2a" (UID: "c2d48667-72e8-463e-866a-ee00e4882c2a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:14:39.581914 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.581884    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-api-access-t87l5" (OuterVolumeSpecName: "kube-api-access-t87l5") pod "c2d48667-72e8-463e-866a-ee00e4882c2a" (UID: "c2d48667-72e8-463e-866a-ee00e4882c2a"). InnerVolumeSpecName "kube-api-access-t87l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:14:39.581914 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.581894    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d48667-72e8-463e-866a-ee00e4882c2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2d48667-72e8-463e-866a-ee00e4882c2a" (UID: "c2d48667-72e8-463e-866a-ee00e4882c2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:14:39.680916 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.680828    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t87l5\" (UniqueName: \"kubernetes.io/projected/c2d48667-72e8-463e-866a-ee00e4882c2a-kube-api-access-t87l5\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:14:39.680916 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.680857    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d48667-72e8-463e-866a-ee00e4882c2a-cabundle-cert\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:14:39.680916 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.680866    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d48667-72e8-463e-866a-ee00e4882c2a-proxy-tls\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:14:39.824074 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.824036    2569 generic.go:356] "Generic (PLEG): container finished" podID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerID="411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246" exitCode=0
Jul 13 19:14:39.824227 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.824106    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"
Jul 13 19:14:39.824227 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.824115    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" event={"ID":"c2d48667-72e8-463e-866a-ee00e4882c2a","Type":"ContainerDied","Data":"411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246"}
Jul 13 19:14:39.824227 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.824182    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865" event={"ID":"c2d48667-72e8-463e-866a-ee00e4882c2a","Type":"ContainerDied","Data":"508cdd0942ee477b0a26865fda79e02d79f39ee2fd7f1b124e17b546593cd5a8"}
Jul 13 19:14:39.824227 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.824208    2569 scope.go:117] "RemoveContainer" containerID="de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592"
Jul 13 19:14:39.832504 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.832484    2569 scope.go:117] "RemoveContainer" containerID="411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246"
Jul 13 19:14:39.839390 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.839375    2569 scope.go:117] "RemoveContainer" containerID="cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742"
Jul 13 19:14:39.846640 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.846612    2569 scope.go:117] "RemoveContainer" containerID="de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592"
Jul 13 19:14:39.846938 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:14:39.846920    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592\": container with ID starting with de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592 not found: ID does not exist" containerID="de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592"
Jul 13 19:14:39.847014 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.846946    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592"} err="failed to get container status \"de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592\": rpc error: code = NotFound desc = could not find container \"de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592\": container with ID starting with de746f809e04cc705cf71fddb638fa18b496c6b9222d95332751e407c0d71592 not found: ID does not exist"
Jul 13 19:14:39.847014 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.846963    2569 scope.go:117] "RemoveContainer" containerID="411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246"
Jul 13 19:14:39.847213 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:14:39.847196    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246\": container with ID starting with 411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246 not found: ID does not exist" containerID="411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246"
Jul 13 19:14:39.847274 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.847219    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246"} err="failed to get container status \"411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246\": rpc error: code = NotFound desc = could not find container \"411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246\": container with ID starting with 411e5a2b3e7d817199471fec653540107a82130126f7d719356afca696457246 not found: ID does not exist"
Jul 13 19:14:39.847274 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.847234    2569 scope.go:117] "RemoveContainer" containerID="cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742"
Jul 13 19:14:39.847472 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:14:39.847456    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742\": container with ID starting with cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742 not found: ID does not exist" containerID="cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742"
Jul 13 19:14:39.847646 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.847476    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742"} err="failed to get container status \"cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742\": rpc error: code = NotFound desc = could not find container \"cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742\": container with ID starting with cf1fb01f93c7d66010f6a2b40d3dfdfe108dde6cbe33ad99a9f1aa7b2a9dd742 not found: ID does not exist"
Jul 13 19:14:39.848191 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.848171    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"]
Jul 13 19:14:39.851241 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:39.851219    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-09af0-predictor-749f6b4655-5d865"]
Jul 13 19:14:40.045459 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:40.045375    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" path="/var/lib/kubelet/pods/c2d48667-72e8-463e-866a-ee00e4882c2a/volumes"
Jul 13 19:14:41.031880 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.031841    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bxp4b/must-gather-74zm6"]
Jul 13 19:14:41.032442 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.032259    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="storage-initializer"
Jul 13 19:14:41.032442 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.032280    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="storage-initializer"
Jul 13 19:14:41.032442 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.032293    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container"
Jul 13 19:14:41.032442 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.032302    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container"
Jul 13 19:14:41.032442 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.032312    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kube-rbac-proxy"
Jul 13 19:14:41.032442 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.032321    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kube-rbac-proxy"
Jul 13 19:14:41.032442 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.032392    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kube-rbac-proxy"
Jul 13 19:14:41.032442 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.032406    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d48667-72e8-463e-866a-ee00e4882c2a" containerName="kserve-container"
Jul 13 19:14:41.037266 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.037243    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:14:41.040269 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.040249    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bxp4b\"/\"openshift-service-ca.crt\""
Jul 13 19:14:41.041269 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.041251    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bxp4b\"/\"kube-root-ca.crt\""
Jul 13 19:14:41.041359 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.041252    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bxp4b\"/\"default-dockercfg-jthcp\""
Jul 13 19:14:41.045357 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.045338    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxp4b/must-gather-74zm6"]
Jul 13 19:14:41.051724 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.051703    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:14:41.090443 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.090416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f4cb620-f576-4099-9e69-b495ea0651c2-must-gather-output\") pod \"must-gather-74zm6\" (UID: \"0f4cb620-f576-4099-9e69-b495ea0651c2\") " pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:14:41.090562 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.090469    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd9tk\" (UniqueName: \"kubernetes.io/projected/0f4cb620-f576-4099-9e69-b495ea0651c2-kube-api-access-fd9tk\") pod \"must-gather-74zm6\" (UID: \"0f4cb620-f576-4099-9e69-b495ea0651c2\") " pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:14:41.125996 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.125973    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:14:41.191180 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.191148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f4cb620-f576-4099-9e69-b495ea0651c2-must-gather-output\") pod \"must-gather-74zm6\" (UID: \"0f4cb620-f576-4099-9e69-b495ea0651c2\") " pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:14:41.191319 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.191212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd9tk\" (UniqueName: \"kubernetes.io/projected/0f4cb620-f576-4099-9e69-b495ea0651c2-kube-api-access-fd9tk\") pod \"must-gather-74zm6\" (UID: \"0f4cb620-f576-4099-9e69-b495ea0651c2\") " pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:14:41.191503 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.191484    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f4cb620-f576-4099-9e69-b495ea0651c2-must-gather-output\") pod \"must-gather-74zm6\" (UID: \"0f4cb620-f576-4099-9e69-b495ea0651c2\") " pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:14:41.202014 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.201987    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd9tk\" (UniqueName: \"kubernetes.io/projected/0f4cb620-f576-4099-9e69-b495ea0651c2-kube-api-access-fd9tk\") pod \"must-gather-74zm6\" (UID: \"0f4cb620-f576-4099-9e69-b495ea0651c2\") " pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:14:41.356628 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.356548    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:14:41.480143 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.480120    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxp4b/must-gather-74zm6"]
Jul 13 19:14:41.482546 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:14:41.482510    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f4cb620_f576_4099_9e69_b495ea0651c2.slice/crio-d0f0a08bbe928e0725bb32dfa2119b047b33489181dbefd2c800b242288538e4 WatchSource:0}: Error finding container d0f0a08bbe928e0725bb32dfa2119b047b33489181dbefd2c800b242288538e4: Status 404 returned error can't find the container with id d0f0a08bbe928e0725bb32dfa2119b047b33489181dbefd2c800b242288538e4
Jul 13 19:14:41.484319 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.484301    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 19:14:41.834345 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:41.834310    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxp4b/must-gather-74zm6" event={"ID":"0f4cb620-f576-4099-9e69-b495ea0651c2","Type":"ContainerStarted","Data":"d0f0a08bbe928e0725bb32dfa2119b047b33489181dbefd2c800b242288538e4"}
Jul 13 19:14:47.866044 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:47.865974    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxp4b/must-gather-74zm6" event={"ID":"0f4cb620-f576-4099-9e69-b495ea0651c2","Type":"ContainerStarted","Data":"6a009418b0235039861431f4a7c6805e77fa4fe6a8f88fb63c5ace05e6204a44"}
Jul 13 19:14:47.866044 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:47.866014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxp4b/must-gather-74zm6" event={"ID":"0f4cb620-f576-4099-9e69-b495ea0651c2","Type":"ContainerStarted","Data":"46e6c124df9f44ce0c75bbcce1ef97f804f606ccf5ee55b3fa1f47a3ca3ab31a"}
Jul 13 19:14:47.885205 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:14:47.885152    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bxp4b/must-gather-74zm6" podStartSLOduration=0.759773718 podStartE2EDuration="6.885136884s" totalImagesPullingTime="6.125363166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:14:47.883969026 +0000 UTC m=+1479.989572647" watchObservedRunningTime="2026-07-13 19:14:47.885136884 +0000 UTC m=+1479.990740503"
Jul 13 19:15:08.950855 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:08.950817    2569 generic.go:356] "Generic (PLEG): container finished" podID="0f4cb620-f576-4099-9e69-b495ea0651c2" containerID="46e6c124df9f44ce0c75bbcce1ef97f804f606ccf5ee55b3fa1f47a3ca3ab31a" exitCode=0
Jul 13 19:15:08.951257 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:08.950891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxp4b/must-gather-74zm6" event={"ID":"0f4cb620-f576-4099-9e69-b495ea0651c2","Type":"ContainerDied","Data":"46e6c124df9f44ce0c75bbcce1ef97f804f606ccf5ee55b3fa1f47a3ca3ab31a"}
Jul 13 19:15:08.951257 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:08.951195    2569 scope.go:117] "RemoveContainer" containerID="46e6c124df9f44ce0c75bbcce1ef97f804f606ccf5ee55b3fa1f47a3ca3ab31a"
Jul 13 19:15:09.220194 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.220112    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bxp4b_must-gather-74zm6_0f4cb620-f576-4099-9e69-b495ea0651c2/gather/0.log"
Jul 13 19:15:09.685396 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.685367    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:09.847972 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.847940    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r9wjh/must-gather-ckxmp"]
Jul 13 19:15:09.850299 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.850284    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9wjh/must-gather-ckxmp"
Jul 13 19:15:09.854926 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.854902    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r9wjh\"/\"default-dockercfg-wwktc\""
Jul 13 19:15:09.855550 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.855384    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9wjh\"/\"openshift-service-ca.crt\""
Jul 13 19:15:09.859385 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.859362    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9wjh\"/\"kube-root-ca.crt\""
Jul 13 19:15:09.876321 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.876294    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9wjh/must-gather-ckxmp"]
Jul 13 19:15:09.901743 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.901715    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:09.945262 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.945183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e1c8073-6798-4630-9da3-660deec8ddb3-must-gather-output\") pod \"must-gather-ckxmp\" (UID: \"0e1c8073-6798-4630-9da3-660deec8ddb3\") " pod="openshift-must-gather-r9wjh/must-gather-ckxmp"
Jul 13 19:15:09.945262 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.945224    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drj5v\" (UniqueName: \"kubernetes.io/projected/0e1c8073-6798-4630-9da3-660deec8ddb3-kube-api-access-drj5v\") pod \"must-gather-ckxmp\" (UID: \"0e1c8073-6798-4630-9da3-660deec8ddb3\") " pod="openshift-must-gather-r9wjh/must-gather-ckxmp"
Jul 13 19:15:09.977995 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.977970    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 19:15:09.978412 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:09.978396    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 19:15:10.028266 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:10.028236    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:10.045832 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:10.045806    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e1c8073-6798-4630-9da3-660deec8ddb3-must-gather-output\") pod \"must-gather-ckxmp\" (UID: \"0e1c8073-6798-4630-9da3-660deec8ddb3\") " pod="openshift-must-gather-r9wjh/must-gather-ckxmp"
Jul 13 19:15:10.045925 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:10.045846    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drj5v\" (UniqueName: \"kubernetes.io/projected/0e1c8073-6798-4630-9da3-660deec8ddb3-kube-api-access-drj5v\") pod \"must-gather-ckxmp\" (UID: \"0e1c8073-6798-4630-9da3-660deec8ddb3\") " pod="openshift-must-gather-r9wjh/must-gather-ckxmp"
Jul 13 19:15:10.046131 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:10.046111    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e1c8073-6798-4630-9da3-660deec8ddb3-must-gather-output\") pod \"must-gather-ckxmp\" (UID: \"0e1c8073-6798-4630-9da3-660deec8ddb3\") " pod="openshift-must-gather-r9wjh/must-gather-ckxmp"
Jul 13 19:15:10.055013 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:10.054992    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drj5v\" (UniqueName: \"kubernetes.io/projected/0e1c8073-6798-4630-9da3-660deec8ddb3-kube-api-access-drj5v\") pod \"must-gather-ckxmp\" (UID: \"0e1c8073-6798-4630-9da3-660deec8ddb3\") " pod="openshift-must-gather-r9wjh/must-gather-ckxmp"
Jul 13 19:15:10.158658 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:10.158624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9wjh/must-gather-ckxmp"
Jul 13 19:15:10.289496 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:10.289468    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9wjh/must-gather-ckxmp"]
Jul 13 19:15:10.292206 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:15:10.292169    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e1c8073_6798_4630_9da3_660deec8ddb3.slice/crio-7b9424b987ae5f0ecfa256fa9cf0d4542fe0ec1acbff4348c5db079af054fcd1 WatchSource:0}: Error finding container 7b9424b987ae5f0ecfa256fa9cf0d4542fe0ec1acbff4348c5db079af054fcd1: Status 404 returned error can't find the container with id 7b9424b987ae5f0ecfa256fa9cf0d4542fe0ec1acbff4348c5db079af054fcd1
Jul 13 19:15:10.959146 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:10.959109    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9wjh/must-gather-ckxmp" event={"ID":"0e1c8073-6798-4630-9da3-660deec8ddb3","Type":"ContainerStarted","Data":"7b9424b987ae5f0ecfa256fa9cf0d4542fe0ec1acbff4348c5db079af054fcd1"}
Jul 13 19:15:11.965173 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:11.965142    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9wjh/must-gather-ckxmp" event={"ID":"0e1c8073-6798-4630-9da3-660deec8ddb3","Type":"ContainerStarted","Data":"b19f0ef39a4586047f88e171164822c9a7969bb32ba26ebca2b23f3238163c0e"}
Jul 13 19:15:11.965634 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:11.965184    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9wjh/must-gather-ckxmp" event={"ID":"0e1c8073-6798-4630-9da3-660deec8ddb3","Type":"ContainerStarted","Data":"7050adfec5215756983e347e03ca6292df0c0f7d0c75db33e149aa9aa55029e9"}
Jul 13 19:15:11.983370 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:11.983313    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r9wjh/must-gather-ckxmp" podStartSLOduration=2.265070869 podStartE2EDuration="2.983292253s" totalImagesPullingTime="718.221384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:15:11.98165249 +0000 UTC m=+1504.087256108" watchObservedRunningTime="2026-07-13 19:15:11.983292253 +0000 UTC m=+1504.088895873"
Jul 13 19:15:12.484368 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:12.484294    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pg7df_03610f49-0413-408e-9e5c-d1adc6d46204/global-pull-secret-syncer/0.log"
Jul 13 19:15:12.540844 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:12.540817    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log"
Jul 13 19:15:12.680098 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:12.680064    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-96.ec2.internal_cfc6c26c84bf408e478864f003148e2a/haproxy/0.log"
Jul 13 19:15:14.662702 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:14.662661    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bxp4b/must-gather-74zm6"]
Jul 13 19:15:14.668146 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:14.662910    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-bxp4b/must-gather-74zm6" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" containerName="copy" containerID="cri-o://6a009418b0235039861431f4a7c6805e77fa4fe6a8f88fb63c5ace05e6204a44" gracePeriod=2
Jul 13 19:15:14.673684 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:14.673655    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bxp4b/must-gather-74zm6"]
Jul 13 19:15:14.986460 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:14.986281    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bxp4b_must-gather-74zm6_0f4cb620-f576-4099-9e69-b495ea0651c2/copy/0.log"
Jul 13 19:15:14.997548 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:14.996119    2569 generic.go:356] "Generic (PLEG): container finished" podID="0f4cb620-f576-4099-9e69-b495ea0651c2" containerID="6a009418b0235039861431f4a7c6805e77fa4fe6a8f88fb63c5ace05e6204a44" exitCode=143
Jul 13 19:15:15.023551 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.023371    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bxp4b_must-gather-74zm6_0f4cb620-f576-4099-9e69-b495ea0651c2/copy/0.log"
Jul 13 19:15:15.027067 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.026826    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:15:15.030120 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:15:15.030077    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-74zm6\" is forbidden: User \"system:node:ip-10-0-130-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bxp4b\": no relationship found between node 'ip-10-0-130-96.ec2.internal' and this object" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:15:15.198443 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.198348    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd9tk\" (UniqueName: \"kubernetes.io/projected/0f4cb620-f576-4099-9e69-b495ea0651c2-kube-api-access-fd9tk\") pod \"0f4cb620-f576-4099-9e69-b495ea0651c2\" (UID: \"0f4cb620-f576-4099-9e69-b495ea0651c2\") "
Jul 13 19:15:15.198443 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.198423    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f4cb620-f576-4099-9e69-b495ea0651c2-must-gather-output\") pod \"0f4cb620-f576-4099-9e69-b495ea0651c2\" (UID: \"0f4cb620-f576-4099-9e69-b495ea0651c2\") "
Jul 13 19:15:15.199944 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.199911    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4cb620-f576-4099-9e69-b495ea0651c2-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0f4cb620-f576-4099-9e69-b495ea0651c2" (UID: "0f4cb620-f576-4099-9e69-b495ea0651c2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:15:15.209118 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.203324    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4cb620-f576-4099-9e69-b495ea0651c2-kube-api-access-fd9tk" (OuterVolumeSpecName: "kube-api-access-fd9tk") pod "0f4cb620-f576-4099-9e69-b495ea0651c2" (UID: "0f4cb620-f576-4099-9e69-b495ea0651c2"). InnerVolumeSpecName "kube-api-access-fd9tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:15:15.299546 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.299493    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd9tk\" (UniqueName: \"kubernetes.io/projected/0f4cb620-f576-4099-9e69-b495ea0651c2-kube-api-access-fd9tk\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:15:15.299546 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.299546    2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f4cb620-f576-4099-9e69-b495ea0651c2-must-gather-output\") on node \"ip-10-0-130-96.ec2.internal\" DevicePath \"\""
Jul 13 19:15:15.770804 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.770581    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:15.872364 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:15.872262    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2l425_7ef7ee3d-6283-41fc-822a-0597c41cdccd/konnectivity-agent/0.log" limit=1024
Jul 13 19:15:16.002677 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:16.002608    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bxp4b_must-gather-74zm6_0f4cb620-f576-4099-9e69-b495ea0651c2/copy/0.log"
Jul 13 19:15:16.007549 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:16.003336    2569 scope.go:117] "RemoveContainer" containerID="6a009418b0235039861431f4a7c6805e77fa4fe6a8f88fb63c5ace05e6204a44"
Jul 13 19:15:16.007549 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:16.003505    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:15:16.011959 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:15:16.011748    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-74zm6\" is forbidden: User \"system:node:ip-10-0-130-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bxp4b\": no relationship found between node 'ip-10-0-130-96.ec2.internal' and this object" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:15:16.022216 ip-10-0-130-96 kubenswrapper[2569]: E0713 19:15:16.022148    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-74zm6\" is forbidden: User \"system:node:ip-10-0-130-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bxp4b\": no relationship found between node 'ip-10-0-130-96.ec2.internal' and this object" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" pod="openshift-must-gather-bxp4b/must-gather-74zm6"
Jul 13 19:15:16.028709 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:16.028685    2569 scope.go:117] "RemoveContainer" containerID="46e6c124df9f44ce0c75bbcce1ef97f804f606ccf5ee55b3fa1f47a3ca3ab31a"
Jul 13 19:15:16.054205 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:16.054164    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" path="/var/lib/kubelet/pods/0f4cb620-f576-4099-9e69-b495ea0651c2/volumes"
Jul 13 19:15:17.365088 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:17.365011    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5rqpt_72182912-d026-4984-acc5-3b9dc462c36b/kube-state-metrics/0.log"
Jul 13 19:15:17.460102 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:17.460073    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5rqpt_72182912-d026-4984-acc5-3b9dc462c36b/kube-rbac-proxy-main/0.log"
Jul 13 19:15:17.498750 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:17.498718    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-5rqpt_72182912-d026-4984-acc5-3b9dc462c36b/kube-rbac-proxy-self/0.log"
Jul 13 19:15:17.890908 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:17.890878    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm5c8_9112d63f-e06b-42ae-8399-a6e195c299af/node-exporter/0.log"
Jul 13 19:15:17.949037 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:17.948937    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm5c8_9112d63f-e06b-42ae-8399-a6e195c299af/kube-rbac-proxy/0.log"
Jul 13 19:15:18.005499 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.005475    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qm5c8_9112d63f-e06b-42ae-8399-a6e195c299af/init-textfile/0.log"
Jul 13 19:15:18.074249 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.074200    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-9pdzz_4340d27b-b543-45f2-a746-eed158a97c28/kube-rbac-proxy-main/0.log"
Jul 13 19:15:18.155185 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.155073    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-9pdzz_4340d27b-b543-45f2-a746-eed158a97c28/kube-rbac-proxy-self/0.log"
Jul 13 19:15:18.200977 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.200908    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-9pdzz_4340d27b-b543-45f2-a746-eed158a97c28/openshift-state-metrics/0.log"
Jul 13 19:15:18.300197 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.300164    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b11fcb47-48c1-4188-809b-035b5deb68d7/prometheus/0.log"
Jul 13 19:15:18.369805 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.369775    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b11fcb47-48c1-4188-809b-035b5deb68d7/config-reloader/0.log"
Jul 13 19:15:18.418483 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.418379    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b11fcb47-48c1-4188-809b-035b5deb68d7/thanos-sidecar/0.log"
Jul 13 19:15:18.464740 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.464714    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b11fcb47-48c1-4188-809b-035b5deb68d7/kube-rbac-proxy-web/0.log"
Jul 13 19:15:18.529827 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.529798    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b11fcb47-48c1-4188-809b-035b5deb68d7/kube-rbac-proxy/0.log"
Jul 13 19:15:18.582237 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.582204    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b11fcb47-48c1-4188-809b-035b5deb68d7/kube-rbac-proxy-thanos/0.log"
Jul 13 19:15:18.642024 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.641992    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b11fcb47-48c1-4188-809b-035b5deb68d7/init-config-reloader/0.log"
Jul 13 19:15:18.698208 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.698128    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6vlsp_83ae5ff5-1970-4f25-99f4-c53c7f19d710/prometheus-operator/0.log"
Jul 13 19:15:18.743013 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.742981    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6vlsp_83ae5ff5-1970-4f25-99f4-c53c7f19d710/kube-rbac-proxy/0.log"
Jul 13 19:15:18.823718 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.823670    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db645677c-8jj4h_bb17b840-15ee-4f40-9a16-264becb9700e/telemeter-client/0.log"
Jul 13 19:15:18.858429 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.858399    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db645677c-8jj4h_bb17b840-15ee-4f40-9a16-264becb9700e/reload/0.log"
Jul 13 19:15:18.906445 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:18.906418    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5db645677c-8jj4h_bb17b840-15ee-4f40-9a16-264becb9700e/kube-rbac-proxy/0.log"
Jul 13 19:15:21.131983 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.131947    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"]
Jul 13 19:15:21.132461 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.132235    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" containerName="gather"
Jul 13 19:15:21.132461 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.132246    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" containerName="gather"
Jul 13 19:15:21.132461 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.132254    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" containerName="copy"
Jul 13 19:15:21.132461 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.132260    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" containerName="copy"
Jul 13 19:15:21.132461 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.132307    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" containerName="gather"
Jul 13 19:15:21.132461 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.132314    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4cb620-f576-4099-9e69-b495ea0651c2" containerName="copy"
Jul 13 19:15:21.136584 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.136554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.146316 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.146289    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"]
Jul 13 19:15:21.156557 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.156495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-sys\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.156717 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.156570    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-podres\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.156717 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.156624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-proc\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.156717 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.156662    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-lib-modules\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.156717 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.156688    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rst7x\" (UniqueName: \"kubernetes.io/projected/aa83191b-c671-4ae2-a16f-449804cf2e3f-kube-api-access-rst7x\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.257600 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.257566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-sys\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.257783 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.257618    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-podres\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.257783 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.257675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-proc\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.257783 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.257695    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-sys\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.257783 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.257714    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-lib-modules\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.257783 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.257762    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rst7x\" (UniqueName: \"kubernetes.io/projected/aa83191b-c671-4ae2-a16f-449804cf2e3f-kube-api-access-rst7x\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.258059 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.257797    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-proc\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.258059 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.257764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-podres\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.258059 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.257838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa83191b-c671-4ae2-a16f-449804cf2e3f-lib-modules\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.267486 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.267455    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rst7x\" (UniqueName: \"kubernetes.io/projected/aa83191b-c671-4ae2-a16f-449804cf2e3f-kube-api-access-rst7x\") pod \"perf-node-gather-daemonset-sbqm5\" (UID: \"aa83191b-c671-4ae2-a16f-449804cf2e3f\") " pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.450684 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.450593    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:21.613072 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:21.613023    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"]
Jul 13 19:15:21.616606 ip-10-0-130-96 kubenswrapper[2569]: W0713 19:15:21.616511    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaa83191b_c671_4ae2_a16f_449804cf2e3f.slice/crio-905045162e3cab121c83708e19915436f389070ba2edecff7fac363304903493 WatchSource:0}: Error finding container 905045162e3cab121c83708e19915436f389070ba2edecff7fac363304903493: Status 404 returned error can't find the container with id 905045162e3cab121c83708e19915436f389070ba2edecff7fac363304903493
Jul 13 19:15:22.038469 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:22.038428    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5" event={"ID":"aa83191b-c671-4ae2-a16f-449804cf2e3f","Type":"ContainerStarted","Data":"e53def8428258f5ed818a7240bc34d60891f916ee3a722bde8ea56fff6eba613"}
Jul 13 19:15:22.038684 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:22.038500    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5" event={"ID":"aa83191b-c671-4ae2-a16f-449804cf2e3f","Type":"ContainerStarted","Data":"905045162e3cab121c83708e19915436f389070ba2edecff7fac363304903493"}
Jul 13 19:15:22.038756 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:22.038714    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:22.064309 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:22.064220    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5" podStartSLOduration=1.064202756 podStartE2EDuration="1.064202756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 19:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:15:22.06385836 +0000 UTC m=+1514.169461994" watchObservedRunningTime="2026-07-13 19:15:22.064202756 +0000 UTC m=+1514.169806375"
Jul 13 19:15:22.457632 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:22.457607    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tz6g6_ec484b05-f937-4909-a863-02e7faa131f9/dns/0.log"
Jul 13 19:15:22.481684 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:22.481659    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tz6g6_ec484b05-f937-4909-a863-02e7faa131f9/kube-rbac-proxy/0.log"
Jul 13 19:15:22.598871 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:22.598839    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z7v8z_7f27934f-42fa-4166-a6ef-eb8062375c7e/dns-node-resolver/0.log"
Jul 13 19:15:23.074058 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:23.074023    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bpjbm_52a0b632-455c-4340-99f2-d4178918e76c/node-ca/0.log"
Jul 13 19:15:24.295572 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:24.295548    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pqhbw_54ed57eb-2265-4c86-9ca9-fbbaa753706f/serve-healthcheck-canary/0.log"
Jul 13 19:15:24.778545 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:24.778501    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngb8f_185e1d28-22e4-4b9f-bf9a-c49a8169e375/kube-rbac-proxy/0.log"
Jul 13 19:15:24.808563 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:24.808519    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngb8f_185e1d28-22e4-4b9f-bf9a-c49a8169e375/exporter/0.log"
Jul 13 19:15:24.832115 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:24.832092    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ngb8f_185e1d28-22e4-4b9f-bf9a-c49a8169e375/extractor/0.log"
Jul 13 19:15:26.981622 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:26.981594    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-swzc7_c2e0b847-946c-4ec7-acfd-a2945b244533/server/0.log"
Jul 13 19:15:27.118875 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:27.118848    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7sntz_6d6a6123-d931-4568-9ab9-14e7981aeb5f/s3-init/0.log"
Jul 13 19:15:27.150355 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:27.150333    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-7sntz_6d6a6123-d931-4568-9ab9-14e7981aeb5f/download-hf-model/0.log"
Jul 13 19:15:28.052033 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:28.052005    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r9wjh/perf-node-gather-daemonset-sbqm5"
Jul 13 19:15:32.477050 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:32.477017    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-864b4_bd43cbf1-95fb-4033-8a3a-bc8d523c80ae/kube-multus/0.log"
Jul 13 19:15:32.779466 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:32.779438    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkdxv_9123bd54-a028-460c-961b-82b16be6168f/kube-multus-additional-cni-plugins/0.log"
Jul 13 19:15:32.798811 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:32.798782    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkdxv_9123bd54-a028-460c-961b-82b16be6168f/egress-router-binary-copy/0.log"
Jul 13 19:15:32.819824 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:32.819803    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkdxv_9123bd54-a028-460c-961b-82b16be6168f/cni-plugins/0.log"
Jul 13 19:15:32.842224 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:32.842195    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkdxv_9123bd54-a028-460c-961b-82b16be6168f/bond-cni-plugin/0.log"
Jul 13 19:15:32.865890 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:32.865868    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkdxv_9123bd54-a028-460c-961b-82b16be6168f/routeoverride-cni/0.log"
Jul 13 19:15:32.885164 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:32.885140    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkdxv_9123bd54-a028-460c-961b-82b16be6168f/whereabouts-cni-bincopy/0.log"
Jul 13 19:15:32.906696 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:32.906664    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nkdxv_9123bd54-a028-460c-961b-82b16be6168f/whereabouts-cni/0.log"
Jul 13 19:15:33.055047 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:33.054967    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pdlbh_c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84/network-metrics-daemon/0.log"
Jul 13 19:15:33.073877 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:33.073816    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pdlbh_c297bb5d-97f3-4da7-8eb0-eb0bab9e5c84/kube-rbac-proxy/0.log"
Jul 13 19:15:34.141219 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:34.141193    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-controller/0.log"
Jul 13 19:15:34.159870 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:34.159843    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/1.log"
Jul 13 19:15:34.160007 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:34.159911    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovn-acl-logging/0.log"
Jul 13 19:15:34.178908 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:34.178886    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/kube-rbac-proxy-node/0.log"
Jul 13 19:15:34.200876 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:34.200854    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/kube-rbac-proxy-ovn-metrics/0.log"
Jul 13 19:15:34.221947 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:34.221926    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/northd/0.log"
Jul 13 19:15:34.241702 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:34.241679    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/nbdb/0.log"
Jul 13 19:15:34.263033 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:34.263015    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/sbdb/0.log"
Jul 13 19:15:34.306968 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:34.306939    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wftcq_7d82fc4d-8d6d-4e5d-a1a1-9258b8b286c7/ovnkube-controller/0.log"
Jul 13 19:15:35.229389 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:35.229354    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5kc4j_8d506c0b-8324-498e-962e-6e396a66159e/network-check-target-container/0.log"
Jul 13 19:15:36.070900 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:36.070872    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-mdpdw_1927b946-784d-4d94-b311-3eb5824fa9ce/iptables-alerter/0.log"
Jul 13 19:15:36.719332 ip-10-0-130-96 kubenswrapper[2569]: I0713 19:15:36.719304    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-599db_103a12f9-a611-452d-ab26-317a59123f8b/tuned/0.log"